[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 492392265 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001013] APIC: Switch to symmetric I/O mode setup [ 0.002259] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007533] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008025] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009021] pid_max: default: 32768 minimum: 301 [ 0.010173] LSM: Security Framework initializing [ 0.011073] Yama: becoming mindful. [ 0.012064] SELinux: Initializing. [ 0.013096] *** VALIDATE selinux *** [ 0.022494] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027593] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028154] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030126] *** VALIDATE tmpfs *** [ 0.031496] *** VALIDATE proc *** [ 0.033107] *** VALIDATE cgroup *** [ 0.034008] *** VALIDATE cgroup2 *** [ 0.035258] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036153] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038023] Spectre V2 : User space: Vulnerable [ 0.039005] Speculative Store Bypass: Vulnerable [ 0.042404] debug: unmapping init [mem 0xffffffff89259000-0xffffffff89260fff] [ 0.045144] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046758] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048016] ... bit width: 48 [ 0.049017] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051020] ... max period: 00007fffffffffff [ 0.052019] ... fixed-purpose events: 3 [ 0.053013] ... event mask: 000000070000000f [ 0.054349] rcu: Hierarchical SRCU implementation. [ 0.056608] smp: Bringing up secondary CPUs ... [ 0.057697] x86: Booting SMP configuration: [ 0.058038] .... node #0, CPUs: #1 #2 #3 [ 0.061587] smp: Brought up 1 node, 4 CPUs [ 0.063015] smpboot: Max logical packages: 1 [ 0.064021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148549] node 0 deferred pages initialised in 79ms [ 0.152110] devtmpfs: initialized [ 0.154272] x86/mm: Memory block size: 128MB [ 0.156854] gcov: version magic: 0x41383552 [ 0.160281] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.164090] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.166332] pinctrl core: initialized pinctrl subsystem [ 0.168180] [ 0.168815] ************************************************************* [ 0.170014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173012] ** ** [ 0.175013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.178013] ** ** [ 0.180011] ** This means that this kernel is built to expose internal ** [ 0.183013] ** IOMMU data structures, which may compromise security on ** [ 0.185010] ** your system. ** [ 0.187013] ** ** [ 0.190014] ** If you see this message and you are not debugging the ** [ 0.192011] ** kernel, report this immediately to your vendor! ** [ 0.195016] ** ** [ 0.197010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.200014] ************************************************************* [ 0.202724] NET: Registered protocol family 16 [ 0.205424] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.208065] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.210067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.214156] cpuidle: using governor menu [ 0.216184] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.217459] PCI: Using configuration type 1 for base access [ 0.218111] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.226171] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.229023] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.233043] cryptd: max_cpu_qlen set to 1000 [ 0.236262] ACPI: Added _OSI(Module Device) [ 0.237013] ACPI: Added _OSI(Processor Device) [ 0.239015] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.241014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.245000] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.252466] ACPI: Interpreter enabled [ 0.254128] ACPI: PM: (supports S0 S3 S4 S5) [ 0.255013] ACPI: Using IOAPIC for interrupt routing [ 0.257121] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.260485] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.270339] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.273034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276025] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.279100] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284302] acpiphp: Slot [2] registered [ 0.286122] acpiphp: Slot [5] registered [ 0.287105] acpiphp: Slot [6] registered [ 0.289109] acpiphp: Slot [7] registered [ 0.290119] acpiphp: Slot [8] registered [ 0.292115] acpiphp: Slot [9] registered [ 0.293137] acpiphp: Slot [10] registered [ 0.295113] acpiphp: Slot [3] registered [ 0.296100] acpiphp: Slot [4] registered [ 0.298096] acpiphp: Slot [11] registered [ 0.300103] acpiphp: Slot [12] registered [ 0.301087] acpiphp: Slot [13] registered [ 0.303137] acpiphp: Slot [14] registered [ 0.304091] acpiphp: Slot [15] registered [ 0.306110] acpiphp: Slot [16] registered [ 0.307105] acpiphp: Slot [17] registered [ 0.309094] acpiphp: Slot [18] registered [ 0.310090] acpiphp: Slot [19] registered [ 0.312107] acpiphp: Slot [20] registered [ 0.313182] acpiphp: Slot [21] registered [ 0.315109] acpiphp: Slot [22] registered [ 0.316121] acpiphp: Slot [23] registered [ 0.318162] acpiphp: Slot [24] registered [ 0.319123] acpiphp: Slot [25] registered [ 0.320115] acpiphp: Slot [26] registered [ 0.322121] acpiphp: Slot [27] registered [ 0.323124] acpiphp: Slot [28] registered [ 0.325121] acpiphp: Slot [29] registered [ 0.326141] acpiphp: Slot [30] registered [ 0.328158] acpiphp: Slot [31] registered [ 0.330069] PCI host bridge to bus 0000:00 [ 0.331022] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.333026] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.336032] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.339024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.341045] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.344043] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.346255] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349250] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.352425] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362834] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.369097] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.372020] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.374024] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.378019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.381359] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.384864] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.388056] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.391853] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.397016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414015] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.421909] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.443016] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.450022] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.498016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.518220] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.528018] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.536085] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.560015] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.577783] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.597017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.605022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.625018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.635717] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.654016] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.666024] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.693019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.707118] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.714016] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.721019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.737021] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.746997] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.757020] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.763017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.792018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.808902] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.814000] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.819031] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.821421] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.824228] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.828165] iommu: Default domain type: Passthrough [ 0.830418] SCSI subsystem initialized [ 0.832198] ACPI: bus type USB registered [ 0.835000] usbcore: registered new interface driver usbfs [ 0.836022] usbcore: registered new interface driver hub [ 0.838173] usbcore: registered new device driver usb [ 0.841180] pps_core: LinuxPPS API ver. 1 registered [ 0.843011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.846108] PTP clock support registered [ 0.849056] EDAC MC: Ver: 3.0.0 [ 0.851124] PCI: Using ACPI for IRQ routing [ 0.852978] NetLabel: Initializing [ 0.854025] NetLabel: domain hash size = 128 [ 0.856010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.858091] NetLabel: unlabeled traffic allowed by default [ 0.860157] vgaarb: loaded [ 0.862275] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.864019] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.871412] clocksource: Switched to clocksource kvm-clock [ 0.986421] VFS: Disk quotas dquot_6.6.0 [ 0.987876] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.990160] *** VALIDATE ramfs *** [ 0.991045] *** VALIDATE hugetlbfs *** [ 0.991958] pnp: PnP ACPI init [ 0.993358] pnp: PnP ACPI: found 6 devices [ 1.008502] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.011224] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.012938] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.014708] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.016684] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.018626] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.021114] NET: Registered protocol family 2 [ 1.023671] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.028796] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.032303] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.037765] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.041981] TCP: Hash tables configured (established 65536 bind 65536) [ 1.045598] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.049264] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.053690] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.056816] NET: Registered protocol family 1 [ 1.059661] RPC: Registered named UNIX socket transport module. [ 1.061526] RPC: Registered udp transport module. [ 1.063247] RPC: Registered tcp transport module. [ 1.065070] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.067443] NET: Registered protocol family 44 [ 1.068486] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.069736] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.071092] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.072505] PCI: CLS 0 bytes, default 64 [ 1.073592] Unpacking initramfs... [ 2.485333] debug: unmapping init [mem 0xffff8f613cc54000-0xffff8f613ffbffff] [ 2.489391] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.491292] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.493468] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.999485] Initialise system trusted keyrings [ 3.001244] Key type blacklist registered [ 3.003233] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.012510] zbud: loaded [ 3.015441] *** VALIDATE nfs *** [ 3.016856] *** VALIDATE nfs4 *** [ 3.019365] pstore: using deflate compression [ 3.023511] Platform Keyring initialized [ 3.132047] NET: Registered protocol family 38 [ 3.133744] Key type asymmetric registered [ 3.135138] Asymmetric key parser 'x509' registered [ 3.136915] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.139926] io scheduler mq-deadline registered [ 3.142032] io scheduler kyber registered [ 3.143654] io scheduler bfq registered [ 3.145589] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.148915] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.151889] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.154786] ACPI: Power Button [PWRF] [ 3.160125] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.166721] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.179829] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.186826] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.205199] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.232362] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.261531] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.267128] Non-volatile memory driver v1.3 [ 3.268945] Linux agpgart interface v0.103 [ 3.303052] virtio_blk virtio1: [vda] 149976 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.305246] vda: detected capacity change from 0 to 76787712 [ 3.320755] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.323740] vdb: detected capacity change from 0 to 1073741824 [ 3.340481] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.343957] vdc: detected capacity change from 0 to 2621440000 [ 3.361892] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.364961] vdd: detected capacity change from 0 to 2621440000 [ 3.376875] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.379849] vde: detected capacity change from 0 to 4294967296 [ 3.393657] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.396890] vdf: detected capacity change from 0 to 4294967296 [ 3.403530] libphy: Fixed MDIO Bus: probed [ 3.412106] usbcore: registered new interface driver usbserial_generic [ 3.414720] usbserial: USB Serial support registered for generic [ 3.417327] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.420485] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.422448] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.424839] mousedev: PS/2 mouse device common for all mice [ 3.427739] rtc_cmos 00:05: RTC can wake from S4 [ 3.431668] rtc_cmos 00:05: registered as rtc0 [ 3.434695] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.435613] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.441449] intel_pstate: CPU model not supported [ 3.443145] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.447890] hid: raw HID events driver (C) Jiri Kosina [ 3.448456] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.449989] usbcore: registered new interface driver usbhid [ 3.455076] usbhid: USB HID core driver [ 3.456575] drop_monitor: Initializing network drop monitor service [ 3.459084] Initializing XFRM netlink socket [ 3.461243] NET: Registered protocol family 10 [ 3.464238] Segment Routing with IPv6 [ 3.465725] NET: Registered protocol family 17 [ 3.467844] mpls_gso: MPLS GSO support [ 3.474067] RAS: Correctable Errors collector initialized. [ 3.476392] AVX version of gcm_enc/dec engaged. [ 3.478170] AES CTR mode by8 optimization enabled [ 3.541622] sched_clock: Marking stable (3541592946, 0)->(4445765996, -904173050) [ 3.544952] registered taskstats version 1 [ 3.547224] Loading compiled-in X.509 certificates [ 3.549229] zswap: loaded using pool lzo/zbud [ 3.572092] Key type big_key registered [ 3.585099] Key type encrypted registered [ 3.586230] ima: No TPM chip found, activating TPM-bypass! [ 3.588524] ima: Allocated hash algorithm: sha1 [ 3.590250] ima: No architecture policies found [ 3.591988] evm: Initialising EVM extended attributes: [ 3.593677] evm: security.selinux [ 3.594866] evm: security.ima [ 3.596068] evm: security.capability [ 3.597393] evm: HMAC attrs: 0x1 [ 3.599746] rtc_cmos 00:05: setting system clock to 2026-09-07 20:42:22 UTC (1788813742) [ 3.606238] debug: unmapping init [mem 0xffffffff8a203000-0xffffffff8a3fffff] [ 3.608971] debug: unmapping init [mem 0xffffffff88f82000-0xffffffff89258fff] [ 3.617084] Write protecting the kernel read-only data: 28672k [ 3.619893] debug: unmapping init [mem 0xffffffff87603000-0xffffffff877fffff] [ 3.622374] debug: unmapping init [mem 0xffffffff87f14000-0xffffffff87ffffff] [ 3.649172] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.658298] systemd[1]: Detected virtualization kvm. [ 3.660157] systemd[1]: Detected architecture x86-64. [ 3.662631] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.686724] systemd[1]: No hostname configured. [ 3.688075] systemd[1]: Set hostname to . [ 3.689419] random: systemd: uninitialized urandom read (16 bytes read) [ 3.691402] systemd[1]: Initializing machine ID from random generator. [ 3.722093] random: ln: uninitialized urandom read (6 bytes read) [ 3.808480] random: systemd: uninitialized urandom read (16 bytes read) [ 3.811282] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.816715] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.823507] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Reached target Initrd Root Device. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.402789] device-mapper: uevent: version 1.0.3 [ 4.404863] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [[ 4.673731] random: fast init done  OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.028585] virtio_net virtio0 ens2: renamed from eth0 [ 5.131449] scsi host0: ata_piix [ 5.176212] scsi host1: ata_piix [ 5.177804] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.180047] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.381637] dracut-initqueue[588]: RTNETLINK answers: File exists [ 10.102555] random: crng init done [ 10.103846] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.442562] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. Stopping udev Kernel Device Manager... [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.646531] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.930085] SELinux: Disabled at runtime. [ 11.999258] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.007981] systemd[1]: Detected virtualization kvm. [ 12.010016] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.566578] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.569528] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.575255] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.578121] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.582266] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.589625] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.595423] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. Mounting Huge Pages File System... [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting POSIX Message Queue File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Activating swap /dev/disk/by-label/SWAP... [ OK ] Created slice system-getty.slice. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... Mounting Kernel Debug File System... [ 12.735763] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting Remount Root and Kernel File Systems... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.178845] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.536557] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.618494] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.689777] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.718830] EDAC sbridge: Ver: 1.1.2 [ 15.556628] Key type dns_resolver registered [ 15.895264] NFS: Registering the id_resolver key type [ 15.897448] Key type id_resolver registered [ 15.899442] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg128-server login: [ 43.592841] libcfs: loading out-of-tree module taints kernel. [ 43.614463] Key type ._llcrypt registered [ 43.616536] Key type .llcrypt registered [ 43.667583] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 52.196899] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 52.984661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 52.995076] alg: No test for adler32 (adler32-zlib) [ 54.058385] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 54.446861] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 56.116386] Key type lgssc registered [ 58.596156] Lustre: Echo OBD driver; http://www.lustre.org/ [ 85.240848] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 132.533988] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 147.332954] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 147.377819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 148.732523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 148.783710] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 148.957924] Lustre: lustre-MDT0000: new disk, initializing [ 149.044626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 149.064513] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 154.430831] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 159.670823] hrtimer: interrupt took 5330558 ns [ 169.809936] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 169.922382] Lustre: 6503:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 169.954596] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 169.959904] Lustre: Skipped 1 previous similar message [ 170.090685] Lustre: lustre-MDT0001: new disk, initializing [ 170.174831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 170.219819] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 170.242052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 175.057882] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 180.711790] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 191.165276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.462885] Lustre: lustre-OST0000: new disk, initializing [ 191.466825] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 191.473463] Lustre: 8442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 191.572830] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.739445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.756305] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.906021] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.108592] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 214.697736] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 214.808891] Lustre: lustre-OST0001: new disk, initializing [ 214.814565] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 214.818578] Lustre: 9515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 214.917444] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 221.312592] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.816216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.833264] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.954784] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.322435] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 240.053842] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 246.404789] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing check_logdir /tmp/testlogs/ [ 251.236783] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing yml_node [ 255.649445] Lustre: DEBUG MARKER: Client: 2.17.58.40 [ 258.330751] Lustre: DEBUG MARKER: MDS: 2.17.58.40 [ 260.855431] Lustre: DEBUG MARKER: OSS: 2.17.58.40 [ 262.469881] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 16:46:39 EDT 2026 [ 279.958648] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 281.275907] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 298.464957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 298.472295] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.490697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.025729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 301.032570] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 302.687891] Lustre: server umount lustre-MDT0000 complete [ 306.546427] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814045 with bad export cookie 15672834520865315127 [ 306.551890] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 306.555550] LustreError: 6497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 311.276864] LustreError: 6512:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 311.277346] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 311.283748] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 311.283757] Lustre: Skipped 3 previous similar messages [ 311.303136] LustreError: 6512:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 311.320028] Lustre: Skipped 3 previous similar messages [ 312.919260] Lustre: server umount lustre-MDT0001 complete [ 316.845376] Lustre: server umount lustre-OST0000 complete [ 320.697490] Lustre: server umount lustre-OST0001 complete [ 327.417787] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 337.137253] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 371.964236] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 372.217294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 372.245483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 372.332558] Lustre: lustre-MDT0000: new disk, initializing [ 372.426667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 372.437476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 376.673138] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 385.142933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 385.249522] Lustre: 15067:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 385.315648] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 385.318828] Lustre: Skipped 1 previous similar message [ 385.376855] Lustre: lustre-MDT0001: new disk, initializing [ 385.471479] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 385.514804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 385.535511] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 389.557616] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 398.245804] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 403.363367] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 409.496954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 409.791938] Lustre: lustre-OST0000: new disk, initializing [ 409.794574] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 409.799102] Lustre: 17036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 409.884805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 411.350979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 411.356785] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 411.485974] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 415.651489] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 423.922492] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 428.167540] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 428.425398] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 432.558792] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 432.754787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 437.225167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 437.226990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 437.242329] LustreError: Skipped 1 previous similar message [ 437.243157] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 437.267807] Lustre: Skipped 1 previous similar message [ 437.283662] Lustre: Skipped 1 previous similar message [ 440.538606] Lustre: server umount lustre-OST0000 complete [ 446.955381] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 446.964914] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.985914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 446.990732] Lustre: Skipped 1 previous similar message [ 449.939368] Lustre: server umount lustre-MDT0000 complete [ 452.579041] LustreError: 15075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.623093] LustreError: 15075:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 453.541646] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814192 with bad export cookie 15672834520865316653 [ 453.548152] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 453.565436] LustreError: 15059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 453.979743] Lustre: server umount lustre-MDT0001 complete [ 457.705681] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 16:49:55 (1788814195) [ 460.075500] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 461.870828] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 16:49:58 (1788814198) [ 468.183127] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.742404] LustreError: 19447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 468.801334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.544487] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 474.086021] LustreError: 19446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 478.181190] LustreError: 19447:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 480.895758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 484.933978] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 490.446542] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 496.079940] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 501.207675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 506.815705] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 513.570459] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 521.187056] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 521.206684] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 521.220353] Lustre: Skipped 1 previous similar message [ 521.229880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 521.232758] Lustre: Skipped 2 previous similar messages [ 525.573566] Lustre: server umount lustre-OST0000 complete [ 534.926103] Lustre: server umount lustre-MDT0000 complete [ 537.834793] LustreError: 19426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814276 with bad export cookie 15672834520865317542 [ 537.837249] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 537.843925] LustreError: 19426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 538.432479] Lustre: server umount lustre-MDT0001 complete [ 549.028417] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 551.715604] Key type lgssc unregistered [ 552.012742] LNet: 22977:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 552.018129] LNetError: 22977:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 552.038992] LNet: Removed LNI 192.168.201.128@tcp [ 553.037326] Key type .llcrypt unregistered [ 553.041362] Key type ._llcrypt unregistered [ 572.097654] Key type ._llcrypt registered [ 572.100191] Key type .llcrypt registered [ 572.241437] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 573.314925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 573.336571] alg: No test for adler32 (adler32-zlib) [ 574.317369] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 574.668697] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 576.360339] Key type lgssc registered [ 577.291744] Lustre: Echo OBD driver; http://www.lustre.org/ [ 583.803838] loop: module loaded [ 583.886898] loop0: detected capacity change from 0 to 204800000 [ 584.075591] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 584.091366] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 584.331673] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 589.382984] loop0: detected capacity change from 0 to 204800000 [ 589.560377] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 589.577328] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 589.585084] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 593.824811] loop0: detected capacity change from 0 to 204800000 [ 598.797130] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 598.826294] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 600.060968] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 600.110329] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 600.204396] Lustre: lustre-MDT0000: new disk, initializing [ 600.272384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 600.295701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 604.696483] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 612.604225] loop1: detected capacity change from 0 to 204800000 [ 617.064814] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 617.314593] Lustre: lustre-OST07c6: new disk, initializing [ 617.320653] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 617.324983] Lustre: Skipped 1 previous similar message [ 617.329742] Lustre: 26135:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 617.393973] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 619.058181] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 619.067437] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 619.098183] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 623.158077] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 627.393761] Lustre: 25121:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 627.402144] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 627.408738] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 627.419930] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 627.432063] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 627.437436] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 643.274643] Lustre: 25121:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 643.280449] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 643.289946] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 643.293888] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 643.306843] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 643.313309] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 648.038902] Lustre: 25122:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 648.044767] Lustre: 25122:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 648.054868] Lustre: 25122:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 648.062837] Lustre: 25122:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 648.074714] Lustre: 25122:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 648.086539] Lustre: 25122:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 648.104575] Lustre: 25122:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 648.122116] Lustre: 25122:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 648.133305] Lustre: 25122:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 648.145663] Lustre: 25122:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 648.159631] Lustre: 25122:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 648.181211] Lustre: 25122:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.366310] Lustre: 25121:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 651.387324] Lustre: 25121:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 651.391935] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 651.397107] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.405773] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 651.410321] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.414375] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 651.419333] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.423854] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 651.429485] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 651.434665] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 651.440131] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 656.140126] Lustre: 25121:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 656.154199] Lustre: 25121:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 656.168716] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 656.172300] Lustre: 25121:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 656.178369] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 656.183840] Lustre: 25121:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 656.190190] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 656.199197] Lustre: 25121:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 656.206000] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 656.210623] Lustre: 25121:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 656.214728] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 656.219753] Lustre: 25121:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 667.108428] Lustre: 25123:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 667.114435] Lustre: 25123:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 667.120509] Lustre: 25123:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 667.124852] Lustre: 25123:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 667.129103] Lustre: 25123:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 667.133020] Lustre: 25123:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 667.136757] Lustre: 25123:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 667.142424] Lustre: 25123:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 667.150679] Lustre: 25123:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 667.156541] Lustre: 25123:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 667.164951] Lustre: 25123:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 667.173194] Lustre: 25123:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 715.514491] Lustre: server umount lustre-OST07c6 complete [ 732.294776] Lustre: server umount lustre-MDT0000 complete [ 746.637372] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 749.065575] Key type lgssc unregistered [ 749.420152] LNet: 28694:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 749.436101] LNetError: 28694:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 749.456870] LNet: Removed LNI 192.168.201.128@tcp [ 750.227565] Key type .llcrypt unregistered [ 750.229801] Key type ._llcrypt unregistered [ 767.416639] Key type ._llcrypt registered [ 767.419186] Key type .llcrypt registered [ 767.540919] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 768.193694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 768.274952] alg: No test for adler32 (adler32-zlib) [ 769.420952] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 769.615157] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 771.256664] Key type lgssc registered [ 772.156640] Lustre: Echo OBD driver; http://www.lustre.org/ [ 778.447071] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 778.474725] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 779.822208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 783.509713] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 789.991292] LustreError: 29959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 790.002771] LustreError: 29959:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 790.088426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 790.382104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 794.195729] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 800.475772] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 804.672522] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 810.025449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 810.322100] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 815.210898] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 821.468776] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 824.693994] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 840.173159] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 840.175665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 840.187571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 840.204067] Lustre: Skipped 1 previous similar message [ 844.128224] Lustre: server umount lustre-OST0000 complete [ 850.405900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 850.412662] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 850.420652] Lustre: Skipped 2 previous similar messages [ 850.428482] Lustre: Skipped 1 previous similar message [ 853.438849] Lustre: server umount lustre-MDT0000 complete [ 855.524542] LustreError: 29954:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 855.540194] LustreError: 29954:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 855.921145] LustreError: 32739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814594 with bad export cookie 17812587152032841715 [ 855.921469] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 855.928329] LustreError: 32739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 856.169430] Lustre: server umount lustre-MDT0001 complete [ 863.369797] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 865.260389] Key type lgssc unregistered [ 865.476651] LNet: 33690:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 865.487019] LNetError: 33690:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 866.539154] LNet: Removed LNI 192.168.201.128@tcp [ 867.255557] Key type .llcrypt unregistered [ 867.258129] Key type ._llcrypt unregistered [ 881.214829] Key type ._llcrypt registered [ 881.220313] Key type .llcrypt registered [ 881.319658] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 881.989918] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 882.005628] alg: No test for adler32 (adler32-zlib) [ 882.980478] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 883.167737] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 884.808761] Key type lgssc registered [ 885.588107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 891.359539] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 891.374880] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 892.731098] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 895.628438] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 901.838699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 902.103726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 905.443508] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 911.220355] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 915.496338] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 920.688448] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 920.937206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 924.923220] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 930.348385] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 938.365698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 952.801748] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 952.816485] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.838637] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.828054] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 953.828904] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 953.829201] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 953.870449] Lustre: Skipped 1 previous similar message [ 955.491438] Lustre: server umount lustre-OST0000 complete [ 958.437425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 958.444185] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 958.450455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.075921] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 964.082915] Lustre: Skipped 1 previous similar message [ 964.857938] Lustre: server umount lustre-MDT0000 complete [ 968.268537] LustreError: 34917:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814707 with bad export cookie 11639568719446281402 [ 968.279877] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 968.655837] Lustre: server umount lustre-MDT0001 complete [ 977.343418] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 979.760680] Key type lgssc unregistered [ 980.024540] LNet: 38490:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 980.028480] LNetError: 38490:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 980.038196] LNet: Removed LNI 192.168.201.128@tcp [ 980.721275] Key type .llcrypt unregistered [ 980.723937] Key type ._llcrypt unregistered [ 997.980760] Key type ._llcrypt registered [ 997.982085] Key type .llcrypt registered [ 998.059453] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 998.873087] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 998.986466] alg: No test for adler32 (adler32-zlib) [ 1000.004883] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 1000.192742] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1001.832283] Key type lgssc registered [ 1002.503428] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1008.434772] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1008.465839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1009.803784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1013.748971] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1019.368504] LustreError: 39882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1019.384118] LustreError: 39882:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1020.679857] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1020.990378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1024.266884] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1029.582790] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1033.720853] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1039.149722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1039.460335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1044.639932] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1050.076958] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1054.827052] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1060.328380] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1060.335258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1060.346358] Lustre: Skipped 2 previous similar messages [ 1060.365330] Lustre: Skipped 2 previous similar messages [ 1061.631415] Lustre: server umount lustre-MDT0000 complete [ 1064.756128] LustreError: 39862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814803 with bad export cookie 6442350921338711946 [ 1064.759036] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1064.764641] LustreError: 39862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1065.061284] Lustre: server umount lustre-MDT0001 complete [ 1076.863402] LustreError: 39862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.28@tcp arrived at 1788814815 with bad export cookie 6442350921338712618 [ 1076.875119] LustreError: 39862:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1094.815442] Lustre: server umount lustre-OST0000 complete [ 1106.903586] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 1108.821586] Key type lgssc unregistered [ 1109.040576] LNet: 43738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1109.045697] LNetError: 43738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1110.120835] LNet: Removed LNI 192.168.201.128@tcp [ 1110.820186] Key type .llcrypt unregistered [ 1110.822564] Key type ._llcrypt unregistered [ 1128.945289] Key type ._llcrypt registered [ 1128.947654] Key type .llcrypt registered [ 1129.035898] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 1130.008749] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1130.044075] alg: No test for adler32 (adler32-zlib) [ 1131.119555] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 1131.331530] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1132.984298] Key type lgssc registered [ 1133.906794] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1139.947208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1139.960879] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.336842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1144.645893] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1151.001347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1151.318342] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1154.516845] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1160.211337] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1164.085764] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1168.234838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1168.442624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1172.445873] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1177.141525] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1187.892291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1191.905294] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1191.915336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.932597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1192.929109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1192.932044] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1192.933150] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1197.911141] Lustre: server umount lustre-OST0000 complete [ 1217.518155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1217.525308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1217.527199] Lustre: Skipped 1 previous similar message [ 1217.548943] Lustre: Skipped 3 previous similar messages [ 1219.796809] Lustre: server umount lustre-MDT0000 complete [ 1223.127520] LustreError: 45124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788814962 with bad export cookie 1726928527309071951 [ 1223.135483] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1223.137893] LustreError: 45124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1229.524601] Lustre: server umount lustre-MDT0001 complete [ 1239.095328] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 1241.439585] Key type lgssc unregistered [ 1241.637647] LNet: 48873:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1241.650113] LNetError: 48873:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1241.669332] LNet: Removed LNI 192.168.201.128@tcp [ 1242.370880] Key type .llcrypt unregistered [ 1242.373562] Key type ._llcrypt unregistered [ 1260.075212] Key type ._llcrypt registered [ 1260.076751] Key type .llcrypt registered [ 1260.168598] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 1261.062212] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1261.093870] alg: No test for adler32 (adler32-zlib) [ 1262.118933] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 1262.317322] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1263.952189] Key type lgssc registered [ 1264.606669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1270.177147] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1270.202326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.573447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1275.434937] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1281.993519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1282.022720] LustreError: 50269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1282.039132] LustreError: 50269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1282.362925] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1285.847225] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1291.045967] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1294.924967] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1299.510768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1299.729215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1304.314635] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1309.295626] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1316.154258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1323.003737] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1325.470414] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1336.807319] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0717h04m34s [ 1337.846483] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1340.243362] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.28@tcp (stopping) [ 1341.300970] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.28@tcp (stopping) [ 1341.925308] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1341.932081] Lustre: Skipped 1 previous similar message [ 1343.014157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1343.021029] Lustre: Skipped 6 previous similar messages [ 1345.512357] Lustre: server umount lustre-MDT0000 complete [ 1346.369503] LustreError: 52974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.244592] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1349.096912] LustreError: 52105:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1349.114093] LustreError: 52105:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1352.510065] LustreError: 52974:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1353.191447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1353.273545] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1353.471235] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.506491] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1354.482639] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1356.530958] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1358.683684] Lustre: DEBUG MARKER: Wait for df (36631) ... [ 1359.556145] Lustre: DEBUG MARKER: done [ 1363.939527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1363.944103] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1363.953282] Lustre: Skipped 1 previous similar message [ 1363.956271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1364.449161] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1367.620978] Lustre: server umount lustre-OST0000 complete [ 1375.204805] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.208079] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1375.212040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1375.212048] Lustre: Skipped 2 previous similar messages [ 1375.213547] Lustre: Skipped 2 previous similar messages [ 1376.444025] Lustre: server umount lustre-MDT0000 complete [ 1378.769695] LustreError: 52913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815117 with bad export cookie 14013342198493198614 [ 1378.773022] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1378.782363] LustreError: 52913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1378.985731] Lustre: server umount lustre-MDT0001 complete [ 1385.276901] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 1387.053212] Key type lgssc unregistered [ 1387.235769] LNet: 55204:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1387.240884] LNetError: 55204:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1388.266016] LNet: Removed LNI 192.168.201.128@tcp [ 1388.753543] Key type .llcrypt unregistered [ 1388.755143] Key type ._llcrypt unregistered [ 1414.363552] Key type ._llcrypt registered [ 1414.364758] Key type .llcrypt registered [ 1414.448199] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 1415.441099] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1415.469093] alg: No test for adler32 (adler32-zlib) [ 1416.459937] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 1416.652828] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1418.288188] Key type lgssc registered [ 1419.149745] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1424.437864] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1424.455408] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.613825] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1425.627539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1425.745520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1429.073157] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1435.183600] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1435.244885] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1435.268066] Lustre: 57384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1435.553443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1438.656976] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1443.624915] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1446.800739] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1450.893049] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1451.014135] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1451.131646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1455.126654] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1457.222983] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1460.108551] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1464.896819] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0717h06m42s [ 1465.814364] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1470.590104] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1472.135870] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1472.165956] LustreError: 57391:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1492.314365] Lustre: lustre-MDT0000: Client bf3c7006-6fbf-4fc7-af2a-20aad30db0de (at 192.168.201.28@tcp) reconnecting [ 1512.176545] LustreError: 57391:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1514.062927] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1515.754861] Lustre: DEBUG MARKER: done [ 1523.687597] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1523.691120] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.692193] LustreError: Skipped 1 previous similar message [ 1523.693362] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1523.706508] Lustre: Skipped 1 previous similar message [ 1526.230929] Lustre: server umount lustre-OST0000 complete [ 1528.804882] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1528.809528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1528.818550] Lustre: Skipped 1 previous similar message [ 1528.822476] Lustre: Skipped 1 previous similar message [ 1529.893633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1529.899814] Lustre: Skipped 2 previous similar messages [ 1534.889712] Lustre: server umount lustre-MDT0000 complete [ 1537.437072] LustreError: 57378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815276 with bad export cookie 17695321038249466494 [ 1537.442171] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1537.449332] LustreError: 57378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1537.710518] Lustre: server umount lustre-MDT0001 complete [ 1543.969853] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 1545.831486] Key type lgssc unregistered [ 1546.034829] LNet: 61716:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1546.041980] LNetError: 61716:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1546.062869] LNet: Removed LNI 192.168.201.128@tcp [ 1546.736845] Key type .llcrypt unregistered [ 1546.741512] Key type ._llcrypt unregistered [ 1574.243393] Key type ._llcrypt registered [ 1574.245142] Key type .llcrypt registered [ 1574.292472] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 1574.962576] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1574.984160] alg: No test for adler32 (adler32-zlib) [ 1575.929188] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 1576.079788] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1577.729420] Key type lgssc registered [ 1578.408587] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1580.556544] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1580.566882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1581.727607] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1581.754188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1581.860197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1582.475778] Lustre: Failing over lustre-MDT0000 [ 1582.644691] Lustre: server umount lustre-MDT0000 complete [ 1586.438062] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:08:44 (1788815324) [ 1590.274154] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1590.529268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1593.130854] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1598.436685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1598.496337] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1598.520207] Lustre: 64485:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1598.683123] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1601.489604] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1606.696333] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1610.513570] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1614.824911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1614.929212] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1615.052609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1618.967705] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1623.114595] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1625.130379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1632.133416] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1639.905678] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1639.910615] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1639.922216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1640.422157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1640.438660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1640.443394] Lustre: Skipped 2 previous similar messages [ 1641.824428] Lustre: server umount lustre-MDT0000 complete [ 1643.943562] LustreError: 64478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815382 with bad export cookie 9398651623583591401 [ 1643.944082] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1643.950211] LustreError: 64478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1644.175932] Lustre: server umount lustre-MDT0001 complete [ 1646.134975] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1648.667915] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1653.874939] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1661.728185] Lustre: 63410:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788815384/real 1788815384] req@00000000e6db6d19 x1875708806728576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788815400 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1661.740206] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1661.747766] Lustre: Skipped 1 previous similar message [ 1664.992665] Lustre: 63412:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788815387/real 1788815387] req@00000000b398bc05 x1875708806728704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788815403 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1666.976164] Lustre: 63413:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788815389/real 1788815389] req@00000000f691b30e x1875708806728832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788815405 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1668.962992] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x826eb7daff5934cc [ 1668.969294] Lustre: MGC192.168.201.128@tcp: Connection restored to 0@lo (at 0@lo) [ 1669.194866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.255258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1670.176727] Lustre: 63411:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788815392/real 1788815392] req@000000006609ecff x1875708806729088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788815408 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1671.565268] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1674.336304] Lustre: 63411:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788815397/real 1788815397] req@00000000457f9222 x1875708806729216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788815413 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1676.033729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1678.600763] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1680.365039] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1682.833312] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1685.997603] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1690.604358] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1690.605813] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1690.609411] Lustre: Skipped 1 previous similar message [ 1695.052647] Lustre: server umount lustre-MDT0000 complete [ 1695.715285] LustreError: 68987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1695.722296] LustreError: 68987:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 1697.005891] LustreError: 64479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815435 with bad export cookie 9398651623583593676 [ 1697.007807] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1697.012362] LustreError: 64479:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1697.159471] Lustre: server umount lustre-MDT0001 complete [ 1700.470498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1705.512373] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.656495] LustreError: 70923:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [ 1706.660141] LustreError: 70923:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@000000006a03cb54 x1875708806750208/t0(0) o250->MGC192.168.201.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788815445 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1706.670370] LustreError: 70923:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [ 1707.180028] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1707.182287] LustreError: 66335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1707.187333] Lustre: Skipped 1 previous similar message [ 1707.201848] LustreError: 66335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1707.250131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1707.253332] Lustre: Skipped 1 previous similar message [ 1707.290316] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1709.413034] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1712.610880] LustreError: 70937:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1712.618713] LustreError: 70937:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1713.842501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1716.303779] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1717.090277] Lustre: 63410:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788815439/real 1788815439] req@000000008b4511eb x1875708806750592/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788815455 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1717.101892] Lustre: 63410:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1719.277455] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1719.282349] Lustre: Skipped 1 previous similar message [ 1720.365821] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1723.503455] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1729.505186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1729.511455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1729.527783] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1729.530689] Lustre: Skipped 2 previous similar messages [ 1732.573376] Lustre: server umount lustre-MDT0000 complete [ 1734.763243] LustreError: 64478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815473 with bad export cookie 9398651623583595055 [ 1734.766340] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1734.773438] LustreError: 64478:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1734.933444] Lustre: server umount lustre-MDT0001 complete [ 1738.202673] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1740.224991] Lustre: DEBUG MARKER: files compared the same [ 1751.755717] Lustre: server umount lustre-OST0000 complete [ 1760.619373] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 1762.318648] Key type lgssc unregistered [ 1762.533715] LNet: 74203:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1762.539132] LNetError: 74203:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1762.556732] LNet: Removed LNI 192.168.201.128@tcp [ 1763.134952] Key type .llcrypt unregistered [ 1763.136847] Key type ._llcrypt unregistered [ 1776.085308] Key type ._llcrypt registered [ 1776.087261] Key type .llcrypt registered [ 1776.151484] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 1776.673544] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1776.685643] alg: No test for adler32 (adler32-zlib) [ 1777.590811] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 1777.721705] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1779.352206] Key type lgssc registered [ 1780.005633] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1784.195447] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1784.207445] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1785.409426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1787.456118] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug +malloc all [ 1792.199389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1792.351755] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1794.911478] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug +malloc all [ 1799.114552] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1802.003824] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1805.512893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1805.677138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1809.109227] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug +malloc all [ 1810.796057] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1812.614300] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1823.201162] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1823.202914] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1823.205060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1823.213776] Lustre: Skipped 1 previous similar message [ 1823.219854] Lustre: Skipped 1 previous similar message [ 1824.013408] Lustre: server umount lustre-OST0000 complete [ 1826.038626] Lustre: server umount lustre-MDT0000 complete [ 1827.806721] LustreError: 78082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815566 with bad export cookie 8753051973679585054 [ 1827.811366] LustreError: 78082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1827.812329] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1827.948333] Lustre: server umount lustre-MDT0001 complete [ 1832.605827] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 1833.969710] Key type lgssc unregistered [ 1834.118733] LNet: 79119:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1834.122356] LNetError: 79119:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1834.131438] LNet: Removed LNI 192.168.201.128@tcp [ 1834.536447] Key type .llcrypt unregistered [ 1834.537882] Key type ._llcrypt unregistered [ 1846.293236] Key type ._llcrypt registered [ 1846.294473] Key type .llcrypt registered [ 1846.344696] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 1846.999075] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1847.019028] alg: No test for adler32 (adler32-zlib) [ 1847.926920] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 1848.063607] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1849.664225] Key type lgssc registered [ 1850.289566] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1854.388862] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1854.397148] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1869.856397] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1874.976300] LustreError: 80483:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.128@tcp: failed processing log, type 4: rc = -110 [ 1901.600148] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1904.464359] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1907.828312] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1911.571600] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1911.636472] Lustre: *** cfs_fail_loc=706, val=0*** [ 1911.804057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1911.846165] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1913.575794] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1916.898955] LustreError: 80509:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1916.909276] LustreError: 80509:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1917.714986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1917.908247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1920.060988] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1923.709687] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1926.087691] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1927.141037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1927.145133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1927.152643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1928.163041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1928.164987] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1928.168993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1933.185992] Lustre: server umount lustre-OST0000 complete [ 1938.403698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1938.404823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1938.405437] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1938.412701] Lustre: Skipped 1 previous similar message [ 1938.414711] Lustre: Skipped 1 previous similar message [ 1941.279723] Lustre: server umount lustre-MDT0000 complete [ 1943.327839] LustreError: 80490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788815682 with bad export cookie 10675023769619324732 [ 1943.332300] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1943.339936] LustreError: 80490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1943.518605] Lustre: server umount lustre-MDT0001 complete [ 1949.204808] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 1950.880456] Key type lgssc unregistered [ 1951.035892] LNet: 84092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1951.039774] LNetError: 84092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1951.052541] LNet: Removed LNI 192.168.201.128@tcp [ 1951.577128] Key type .llcrypt unregistered [ 1951.578057] Key type ._llcrypt unregistered [ 1965.102969] Key type ._llcrypt registered [ 1965.104397] Key type .llcrypt registered [ 1965.149163] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 1965.568912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1965.620845] alg: No test for adler32 (adler32-zlib) [ 1966.544069] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 1966.666345] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 1968.288198] Key type lgssc registered [ 1968.948459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1972.914453] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1972.935790] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1975.990616] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1979.504733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.811238] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1981.841734] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1984.993287] LustreError: 86168:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1985.008718] LustreError: 86168:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1986.541494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1986.728760] LustreError: 86166:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1986.749956] LustreError: 86166:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1986.752444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1989.859244] LustreError: 86171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1989.867871] LustreError: 86171:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1990.037566] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 1994.880508] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1994.891670] Lustre: MGS: Client 187d8a39-cdf4-4713-9d45-d52a7615e293 (at 0@lo) reconnecting [ 1994.905077] LustreError: 86182:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1995.062882] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1995.126632] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1997.478629] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2007.522039] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2007.526096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2007.536497] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2010.595646] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2010.596331] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2010.601793] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2013.378830] Lustre: server umount lustre-OST0000 complete [ 2015.713534] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.718989] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2015.721679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.721686] Lustre: Skipped 1 previous similar message [ 2020.835070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.836723] Lustre: Skipped 2 previous similar messages [ 2021.593887] Lustre: server umount lustre-MDT0000 complete [ 2023.920327] Lustre: server umount lustre-MDT0001 complete [ 2025.910475] Lustre: server umount MGS complete [ 2032.227726] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:16:10 (1788815770) [ 2049.538120] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 2055.204753] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 2073.823060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.103923] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2081.676050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.796149] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2081.817371] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2081.857591] Lustre: lustre-MDT0001: new disk, initializing [ 2081.900286] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2081.913183] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2081.916305] Lustre: 93326:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2081.920702] LustreError: 93326:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2083.810493] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2089.448758] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2089.600595] Lustre: lustre-OST0000: new disk, initializing [ 2089.604573] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2089.607880] Lustre: 94091:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2089.645601] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2092.682344] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2094.243822] Lustre: 94498:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003ebfdbdd x1875709216721536/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788815849 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2097.778578] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2101.267641] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2101.274836] Lustre: MGS: Client 968e4e2b-5a7d-4242-b79d-0880b9a9e0f1 (at 0@lo) reconnecting [ 2101.300988] Lustre: 92595:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2101.316665] LustreError: 93312:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2101.321837] LustreError: 93312:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2101.325528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2101.367798] Lustre: lustre-MDT0000: new disk, initializing [ 2101.463840] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2101.479354] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2102.502111] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2102.507838] Lustre: 95160:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000ff28b803 x1875709216727552/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788815857 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2103.714756] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2107.564348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2107.569062] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2107.634336] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2132.962936] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.965196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2132.981389] Lustre: Skipped 2 previous similar messages [ 2132.988033] Lustre: Skipped 1 previous similar message [ 2135.214765] Lustre: server umount lustre-OST0000 complete [ 2138.081449] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2138.083560] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.100325] Lustre: Skipped 1 previous similar message [ 2143.201695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2143.205375] Lustre: Skipped 3 previous similar messages [ 2143.577288] Lustre: server umount lustre-MDT0000 complete [ 2151.576608] Lustre: server umount lustre-MDT0001 complete [ 2153.401609] Lustre: server umount MGS complete [ 2158.650936] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:18:16 (1788815896) [ 2168.143697] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2169.623811] Key type lgssc unregistered [ 2169.798603] LNet: 97653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2169.805952] LNetError: 97653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2169.818113] LNet: Removed LNI 192.168.201.128@tcp [ 2170.295226] Key type .llcrypt unregistered [ 2170.297100] Key type ._llcrypt unregistered [ 2179.834191] Key type ._llcrypt registered [ 2179.835217] Key type .llcrypt registered [ 2179.893024] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 2180.439979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2180.468863] alg: No test for adler32 (adler32-zlib) [ 2181.377363] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 2181.535286] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 2183.176270] Key type lgssc registered [ 2183.894768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2188.171605] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2188.187264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.442348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2191.699459] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2196.848503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2197.103555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2199.800653] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2204.086269] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2206.791488] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2210.540564] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2210.793104] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.368873] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2214.912500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2223.089709] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2236.387178] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2236.390742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2236.414096] Lustre: Skipped 1 previous similar message [ 2236.419487] Lustre: Skipped 1 previous similar message [ 2241.530936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2241.555673] Lustre: Skipped 1 previous similar message [ 2242.651468] Lustre: server umount lustre-OST0000 complete [ 2251.745506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2251.766949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2255.782744] Lustre: server umount lustre-MDT0000 complete [ 2256.882807] LustreError: 98849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2256.911446] LustreError: 98849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2261.629919] LustreError: 98829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816000 with bad export cookie 2138895942177039043 [ 2261.640670] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2262.777983] Lustre: server umount lustre-MDT0001 complete [ 2283.923631] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2290.041675] Key type lgssc unregistered [ 2290.704852] LNet: 102407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2290.710359] LNetError: 102407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2290.734910] LNet: Removed LNI 192.168.201.128@tcp [ 2291.820567] Key type .llcrypt unregistered [ 2291.822091] Key type ._llcrypt unregistered [ 2309.933296] Key type ._llcrypt registered [ 2309.936694] Key type .llcrypt registered [ 2310.052675] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 2311.038569] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2311.117897] alg: No test for adler32 (adler32-zlib) [ 2312.229775] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 2312.471773] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 2314.152233] Key type lgssc registered [ 2315.425826] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2321.983385] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2321.992394] LustreError: 103481:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2322.017524] LustreError: 103488:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2322.021843] LustreError: 103481:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4975 [ 2322.025578] LustreError: 103481:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2322.034708] LustreError: 103481:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2322.045162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2323.701318] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2331.828208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.215532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2336.317300] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2341.339756] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing lsmod [ 2343.716889] LustreError: 104683:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 2343.771867] LustreError: 104693:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 2343.777773] LustreError: 104683:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4944 [ 2343.785739] LustreError: 104683:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2343.793535] LustreError: 104683:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2343.797605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2344.163340] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2347.499981] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2347.508935] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2347.526940] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.541226] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2349.556541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.562287] Lustre: Skipped 1 previous similar message [ 2349.598170] Lustre: Skipped 1 previous similar message [ 2354.658341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2354.670207] Lustre: Skipped 3 previous similar messages [ 2357.738240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2357.744198] Lustre: Skipped 2 previous similar messages [ 2361.312243] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2361.603439] Lustre: server umount lustre-MDT0000 complete [ 2362.851231] LustreError: 103525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2362.862870] LustreError: 103525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2365.151583] LustreError: 103506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816104 with bad export cookie 16130504997397226509 [ 2365.157938] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2365.166174] LustreError: 103506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2365.984451] LustreError: 103955:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000c9708d4d x1875709579514240/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2366.111301] Lustre: server umount lustre-MDT0001 complete [ 2380.003517] Lustre: server umount lustre-OST0000 complete [ 2387.673183] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2388.040198] LustreError: 105758:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.110492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2392.257333] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2393.577881] LustreError: 105759:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2397.666778] LustreError: 105758:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2400.508790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2400.954442] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2405.646148] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2412.875398] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2417.972116] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2424.000168] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2424.308615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2430.078824] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2432.532829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2436.663676] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2445.706664] Lustre: server umount lustre-OST0000 complete [ 2451.938617] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2451.946406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2451.962753] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2461.152249] Lustre: 102889:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788816184/real 1788816184] req@00000000a5202a7d x1875709579541760/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788816200 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2461.184114] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2461.208581] Lustre: Skipped 1 previous similar message [ 2461.221339] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2461.231070] Lustre: Skipped 5 previous similar messages [ 2463.201391] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2463.203511] Lustre: 102887:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788816186/real 1788816186] req@0000000007916495 x1875709579542528/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788816202 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2463.415642] Lustre: server umount lustre-MDT0000 complete [ 2466.277330] LustreError: 105758:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2466.618281] LustreError: 105738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816205 with bad export cookie 16130504997397227195 [ 2466.621397] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2466.628323] LustreError: 105738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2466.834117] Lustre: server umount lustre-MDT0001 complete [ 2477.495122] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2480.296401] Key type lgssc unregistered [ 2480.624986] LNet: 109288:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2480.643928] LNetError: 109288:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2480.680724] LNet: Removed LNI 192.168.201.128@tcp [ 2481.620924] Key type .llcrypt unregistered [ 2481.623479] Key type ._llcrypt unregistered [ 2502.625373] Key type ._llcrypt registered [ 2502.627902] Key type .llcrypt registered [ 2502.737300] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 2503.654838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2503.718560] alg: No test for adler32 (adler32-zlib) [ 2504.732532] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 2504.939404] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 2506.648506] Key type lgssc registered [ 2507.565089] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2514.746303] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2514.758155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2516.259588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2520.978830] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2526.691402] LustreError: 110679:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2526.717582] LustreError: 110679:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2527.715524] LustreError: 110680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.988605] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2529.450277] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2533.880914] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2540.569296] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2545.730344] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2551.748921] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2552.025806] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2557.793989] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2562.292445] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2565.463331] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2571.299825] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2574.459827] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2580.178935] Lustre: server umount lustre-OST0000 complete [ 2586.081417] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2586.089704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2586.107484] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2588.134825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2588.136314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2593.251763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2593.258646] Lustre: Skipped 2 previous similar messages [ 2596.320159] Lustre: 109990:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788816319/real 1788816319] req@000000007f9ccf43 x1875709780853376/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788816335 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2596.345930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2596.361273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2598.368376] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2598.436381] Lustre: 109993:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788816321/real 1788816321] req@000000005c2da1b2 x1875709780854272/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788816337 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2598.693287] Lustre: server umount lustre-MDT0000 complete [ 2601.444059] LustreError: 110680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.454072] LustreError: 110680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2602.708875] LustreError: 110659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816341 with bad export cookie 15639281867290795561 [ 2602.716396] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2602.720597] LustreError: 110659:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2603.166844] Lustre: server umount lustre-MDT0001 complete [ 2614.153452] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2616.895357] Key type lgssc unregistered [ 2617.218822] LNet: 114317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2617.223232] LNetError: 114317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2618.281982] LNet: Removed LNI 192.168.201.128@tcp [ 2619.369308] Key type .llcrypt unregistered [ 2619.371518] Key type ._llcrypt unregistered [ 2636.333636] Key type ._llcrypt registered [ 2636.336484] Key type .llcrypt registered [ 2636.433533] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 2637.340874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2637.398091] alg: No test for adler32 (adler32-zlib) [ 2638.457374] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 2638.669974] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 2640.320386] Key type lgssc registered [ 2641.731886] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2649.455665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2649.513875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2650.994498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2655.494512] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2660.324997] LustreError: 115556:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2660.348282] LustreError: 115556:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2664.750442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2665.015770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2669.743525] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2676.996521] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2682.875647] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2689.748712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2690.168621] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2695.288808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2696.719231] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2704.559341] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2716.134440] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2716.149311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.174825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2720.231380] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2720.233382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2720.244304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2721.846790] Lustre: server umount lustre-OST0000 complete [ 2725.345912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2725.350542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2725.368390] Lustre: Skipped 1 previous similar message [ 2725.383464] Lustre: Skipped 2 previous similar messages [ 2730.469119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2730.471818] Lustre: Skipped 1 previous similar message [ 2731.475562] Lustre: server umount lustre-MDT0000 complete [ 2735.118749] LustreError: 116292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816474 with bad export cookie 17495636241210480854 [ 2735.127187] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2735.132602] LustreError: 116292:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2735.647169] Lustre: server umount lustre-MDT0001 complete [ 2747.546810] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2750.232115] Key type lgssc unregistered [ 2750.558582] LNet: 119093:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2750.567579] LNetError: 119093:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2750.584987] LNet: Removed LNI 192.168.201.128@tcp [ 2751.684161] Key type .llcrypt unregistered [ 2751.688082] Key type ._llcrypt unregistered [ 2776.194921] Key type ._llcrypt registered [ 2776.197994] Key type .llcrypt registered [ 2776.274475] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 2777.368681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2777.495667] alg: No test for adler32 (adler32-zlib) [ 2778.718771] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 2778.937848] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 2780.584312] Key type lgssc registered [ 2781.779738] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2789.106468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2789.119240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2790.597062] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2794.698501] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2801.122644] LustreError: 120511:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2801.138617] LustreError: 120511:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2802.491996] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.881538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2806.961451] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2813.877216] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2818.921827] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2825.067632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2825.444143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2830.882906] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2836.739958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2838.849729] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2846.087966] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2849.586993] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2849.596102] Lustre: Skipped 2 previous similar messages [ 2853.462506] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 2853.488213] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2853.615744] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2853.620795] Lustre: Skipped 2 previous similar messages [ 2862.136994] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2862.142762] Lustre: Skipped 1 previous similar message [ 2867.307471] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.28@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2867.315268] Lustre: Skipped 2 previous similar messages [ 2868.319712] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.28@tcp 192.168.0.[2,10]@tcp [ 2880.463506] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.28/24@tcp in log lustre-MDT0000 [ 2880.468868] Lustre: Skipped 2 previous similar messages [ 2888.253943] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.28/24@tcp [ 2888.266376] Lustre: Skipped 1 previous similar message [ 2893.282383] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2893.289407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2893.304826] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2893.310240] Lustre: Skipped 1 previous similar message [ 2898.344474] Lustre: server umount lustre-OST0000 complete [ 2903.522792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2903.529457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2903.536828] Lustre: Skipped 2 previous similar messages [ 2903.546342] Lustre: Skipped 1 previous similar message [ 2907.396241] Lustre: server umount lustre-MDT0000 complete [ 2908.649074] LustreError: 120506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.681649] LustreError: 120506:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2910.293032] LustreError: 120491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816649 with bad export cookie 7347676097496692316 [ 2910.293032] LustreError: 120493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816649 with bad export cookie 7347676097496692316 [ 2910.294645] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2910.299787] LustreError: 120491:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2910.516522] Lustre: server umount lustre-MDT0001 complete [ 2918.984966] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 2920.994367] Key type lgssc unregistered [ 2921.196767] LNet: 125549:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2921.201395] LNetError: 125549:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2921.211923] LNet: Removed LNI 192.168.201.128@tcp [ 2921.863069] Key type .llcrypt unregistered [ 2921.865281] Key type ._llcrypt unregistered [ 2942.725457] Key type ._llcrypt registered [ 2942.726510] Key type .llcrypt registered [ 2942.810577] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 2943.591174] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2943.622579] alg: No test for adler32 (adler32-zlib) [ 2944.623622] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 2944.811802] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 2946.464285] Key type lgssc registered [ 2947.355278] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2952.749805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2952.766156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2954.058198] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2954.084138] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.28/24@tcp [ 2954.100258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2956.916482] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2962.981692] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2963.234624] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2963.249897] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.28/24@tcp [ 2963.290259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2966.900723] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2972.406885] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2977.043864] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2981.946796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2982.178357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.643439] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 2992.795435] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2999.602099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3008.995713] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3009.000517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.008724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3010.026143] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3010.029709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3010.046501] Lustre: Skipped 1 previous similar message [ 3011.847282] Lustre: server umount lustre-OST0000 complete [ 3015.141698] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3015.148811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3015.151545] Lustre: Skipped 1 previous similar message [ 3015.165745] Lustre: Skipped 1 previous similar message [ 3020.257119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3020.269387] Lustre: Skipped 1 previous similar message [ 3021.184885] Lustre: server umount lustre-MDT0000 complete [ 3024.202471] LustreError: 127067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816763 with bad export cookie 3381552250470484149 [ 3024.206992] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3024.216364] LustreError: 127067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3024.457358] Lustre: server umount lustre-MDT0001 complete [ 3032.578548] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 3035.154645] Key type lgssc unregistered [ 3035.438487] LNet: 130668:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3035.448842] LNetError: 130668:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3035.474953] LNet: Removed LNI 192.168.201.128@tcp [ 3036.460587] Key type .llcrypt unregistered [ 3036.466322] Key type ._llcrypt unregistered [ 3059.045316] Key type ._llcrypt registered [ 3059.051844] Key type .llcrypt registered [ 3059.144581] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 3071.750712] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3072.710169] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3072.725102] alg: No test for adler32 (adler32-zlib) [ 3073.700712] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 3073.850486] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 3075.528428] Key type lgssc registered [ 3076.600804] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3103.159581] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3103.180245] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3104.393625] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3104.428249] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3104.485367] Lustre: lustre-MDT0000: new disk, initializing [ 3104.554461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3104.573461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3108.118480] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3116.945094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3117.029517] Lustre: 134044:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3117.069258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3117.078086] Lustre: Skipped 1 previous similar message [ 3117.148696] Lustre: lustre-MDT0001: new disk, initializing [ 3117.226166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3117.265589] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3117.286771] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3121.461753] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3128.959683] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3133.150070] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3138.343227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3138.518484] Lustre: lustre-OST0000: new disk, initializing [ 3138.526070] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3138.531546] Lustre: 136010:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3138.602631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3143.693383] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3147.820295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3147.834443] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3147.960294] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3151.414862] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3155.041158] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3155.197252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3158.956140] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3159.137732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3163.620918] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3163.622954] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3163.625569] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3163.633295] Lustre: Skipped 1 previous similar message [ 3163.641442] LustreError: Skipped 1 previous similar message [ 3166.768458] Lustre: server umount lustre-OST0000 complete [ 3173.858713] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3173.862347] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3173.873250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3173.873260] Lustre: Skipped 1 previous similar message [ 3175.633679] Lustre: server umount lustre-MDT0000 complete [ 3178.366587] LustreError: 136009:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788816917 with bad export cookie 11706944482844888202 [ 3178.385654] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3178.650078] Lustre: server umount lustre-MDT0001 complete [ 3185.187084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3185.653285] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3188.516778] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3190.757371] LustreError: 138226:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3190.778324] LustreError: 138226:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3194.927798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3195.214476] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3198.688707] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3203.791620] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3207.552522] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3212.595387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3212.782294] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3217.191725] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3222.645406] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3226.393847] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3231.884246] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 3234.998536] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3235.168131] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3242.352129] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3242.465659] Lustre: lustre-OST0001: new disk, initializing [ 3242.469898] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3242.481167] Lustre: 141633:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3242.536354] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3246.325402] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3249.742996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3249.749699] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3249.813139] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3252.460264] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3252.631785] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3255.834783] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3255.990056] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3259.758332] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f938794b800.ost_server_uuid 50 [ 3260.659946] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f938794b800.ost_server_uuid in FULL state after 0 sec [ 3270.629889] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3270.641299] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3270.643321] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3270.649950] Lustre: Skipped 2 previous similar messages [ 3270.654165] Lustre: Skipped 1 previous similar message [ 3272.690318] Lustre: server umount lustre-OST0001 complete [ 3275.764341] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3275.770943] LustreError: 140049:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3275.771249] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3275.795517] Lustre: Skipped 1 previous similar message [ 3275.831528] LustreError: 140049:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3280.871928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3280.876736] LustreError: 142933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3280.881567] Lustre: Skipped 2 previous similar messages [ 3280.895401] LustreError: 142933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3281.704159] Lustre: server umount lustre-OST0000 complete [ 3285.989395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.300310] Lustre: server umount lustre-MDT0000 complete [ 3291.106380] LustreError: 138225:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3291.119311] LustreError: 138225:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3292.645720] LustreError: 138205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817031 with bad export cookie 11706944482844889084 [ 3292.648074] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3292.656441] LustreError: 138205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 3292.827557] Lustre: server umount lustre-MDT0001 complete [ 3306.331396] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 3308.329966] Key type lgssc unregistered [ 3308.598185] LNet: 144630:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3308.603897] LNetError: 144630:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3308.624268] LNet: Removed LNI 192.168.201.128@tcp [ 3309.368220] Key type .llcrypt unregistered [ 3309.369970] Key type ._llcrypt unregistered [ 3333.460431] Key type ._llcrypt registered [ 3333.462366] Key type .llcrypt registered [ 3333.550567] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 3343.177440] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3343.921822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3343.935476] alg: No test for adler32 (adler32-zlib) [ 3344.877266] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 3345.036301] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 3346.672224] Key type lgssc registered [ 3347.368150] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3369.437612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3369.458328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3370.619835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3370.642811] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3370.710816] Lustre: lustre-MDT0000: new disk, initializing [ 3370.769330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3370.779587] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3373.139514] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3380.273622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3380.369935] Lustre: 148752:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3380.413298] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3380.416809] Lustre: Skipped 1 previous similar message [ 3380.468953] Lustre: lustre-MDT0001: new disk, initializing [ 3380.561395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3380.591707] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3380.602471] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3383.458313] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3390.311039] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3393.590546] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3397.581769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3397.777129] Lustre: lustre-OST0000: new disk, initializing [ 3397.781727] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3397.784988] Lustre: 150723:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3397.844273] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3401.325306] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3404.823391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3404.831583] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3404.885183] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3407.548802] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3410.377316] Lustre: Failing over lustre-OST0000 [ 3410.482843] Lustre: server umount lustre-OST0000 complete [ 3425.105400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3425.251667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3425.355411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3425.366828] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3426.004283] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3428.733614] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3429.856901] Lustre: 146978:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788817152/real 1788817152] req@0000000018d35f60 x1875710661653760/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788817168 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3430.394667] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3430.395901] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 3430.950728] Lustre: Failing over lustre-MDT0000 [ 3431.134697] Lustre: server umount lustre-MDT0000 complete [ 3431.760548] LustreError: 148761:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3431.774722] LustreError: 148761:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3431.905562] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3431.911602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3431.919578] Lustre: Skipped 1 previous similar message [ 3431.968245] Lustre: 146981:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788817153/real 1788817153] req@000000002816a7f8 x1875710661654656/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788817169 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3435.490223] LustreError: 149886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3435.491572] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3435.500151] LustreError: 149886:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3435.505790] Lustre: Skipped 1 previous similar message [ 3436.512252] Lustre: 146978:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788817159/real 1788817159] req@00000000a575ab0a x1875710661655808/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788817175 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3436.857407] LustreError: 148760:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3439.859874] LustreError: 148759:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3445.052411] LustreError: 150129:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3445.063956] LustreError: 150129:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 3445.665972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3445.756459] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3445.764146] LustreError: 152948:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGC192.168.201.128@tcp: namespace resource [0x65727473756c:0x2:0x0].0x0 (00000000686bc3aa) refcount nonzero (1) after lock cleanup; forcing cleanup. [ 3445.777632] Lustre: 148757:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.201.128@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3445.913314] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3445.944423] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3448.009226] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3448.332832] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3451.369103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3451.375084] Lustre: Skipped 1 previous similar message [ 3451.382773] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 3451.420323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3455.982872] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3455.994820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3456.005955] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3460.080747] Lustre: server umount lustre-OST0000 complete [ 3466.723240] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3466.725669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3466.731331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3466.731338] LustreError: Skipped 1 previous similar message [ 3466.736410] Lustre: Skipped 2 previous similar messages [ 3466.748462] Lustre: Skipped 3 previous similar messages [ 3468.331965] Lustre: server umount lustre-MDT0000 complete [ 3470.563424] LustreError: 148745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817209 with bad export cookie 5969134126667314298 [ 3470.571563] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3470.573442] LustreError: 148745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3470.781245] Lustre: server umount lustre-MDT0001 complete [ 3476.803548] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 3478.451797] Key type lgssc unregistered [ 3478.610556] LNet: 154375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3478.614148] LNetError: 154375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3478.633065] LNet: Removed LNI 192.168.201.128@tcp [ 3479.156458] Key type .llcrypt unregistered [ 3479.158695] Key type ._llcrypt unregistered [ 3504.139065] Key type ._llcrypt registered [ 3504.140695] Key type .llcrypt registered [ 3504.207489] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3504.805160] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3504.836319] alg: No test for adler32 (adler32-zlib) [ 3505.779797] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 3505.902513] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 3507.520177] Key type lgssc registered [ 3508.176536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3512.952711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3512.966027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3514.216472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3516.653324] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3522.264883] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3524.585892] LustreError: 155889:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3524.595916] LustreError: 155889:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3525.016493] LustreError: 155917:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3525.020665] LustreError: 155917:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3525.024520] LustreError: 155917:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 10, retries 0, failed: rc = -5 [ 3525.196430] Lustre: server umount lustre-MDT0000 complete [ 3533.607867] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 3538.589843] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3559.136657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3559.251913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3559.270071] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3559.316310] Lustre: lustre-MDT0000: new disk, initializing [ 3559.349107] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3559.356952] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3561.220709] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3566.767060] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.846122] Lustre: 159217:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3566.883779] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3566.885722] Lustre: Skipped 1 previous similar message [ 3566.925388] Lustre: lustre-MDT0001: new disk, initializing [ 3566.993316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3567.026696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3567.038369] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3569.657150] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3574.814301] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3577.407155] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3580.869802] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3580.987379] Lustre: lustre-OST0000: new disk, initializing [ 3580.989343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3580.992474] Lustre: 161184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3581.030574] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3584.651567] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3589.833565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3589.865686] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3590.193180] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3591.693653] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3596.865224] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3600.077542] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 17:42:17 (1788817337) [ 3820.513204] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3820.516011] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3820.529907] Lustre: Skipped 2 previous similar messages [ 3820.530066] Lustre: Skipped 1 previous similar message [ 3825.636763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3825.646615] Lustre: Skipped 3 previous similar messages [ 3825.847523] Lustre: server umount lustre-MDT0000 complete [ 3829.193268] LustreError: 160013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817568 with bad export cookie 4866665977674697857 [ 3829.193949] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3829.200042] LustreError: 160013:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3830.753670] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3830.755916] LustreError: 159224:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3830.767979] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3834.788553] LustreError: 159222:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3834.790362] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3835.615847] Lustre: server umount lustre-MDT0001 complete [ 3845.303370] Lustre: server umount lustre-OST0000 complete [ 3855.199244] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 3866.861923] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 3901.457299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3901.721215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3901.745739] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3901.810363] Lustre: lustre-MDT0000: new disk, initializing [ 3901.882056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3901.896291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3906.582533] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3916.198488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.312948] Lustre: 165841:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3916.352874] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3916.355984] Lustre: Skipped 1 previous similar message [ 3916.414311] Lustre: lustre-MDT0001: new disk, initializing [ 3916.503200] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3916.534915] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3916.547688] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3920.836966] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3930.396738] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3935.811888] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3941.933255] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3942.127950] Lustre: lustre-OST0000: new disk, initializing [ 3942.129782] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3942.132868] Lustre: 167810:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3942.214423] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3947.578352] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 3949.101603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3949.113875] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3949.228573] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3957.474263] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3960.979752] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3961.120968] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3966.261217] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3966.436950] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3969.508077] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3969.523714] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3969.571813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3970.021135] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3980.270900] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3980.273491] Lustre: Skipped 5 previous similar messages [ 3982.304302] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3982.723482] Lustre: server umount lustre-OST0000 complete [ 3988.449547] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3988.453940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3988.464764] Lustre: Skipped 1 previous similar message [ 4000.739457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4000.762916] Lustre: Skipped 6 previous similar messages [ 4001.249734] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4001.496707] Lustre: server umount lustre-MDT0000 complete [ 4005.702801] LustreError: 165833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817744 with bad export cookie 4866665977674856680 [ 4005.703232] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4005.749392] LustreError: 165833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4006.033803] Lustre: server umount lustre-MDT0001 complete [ 4014.250818] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:49:11 (1788817751) [ 4030.866549] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 4041.466526] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 4077.670386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4077.900305] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4077.923461] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4077.999605] Lustre: lustre-MDT0000: new disk, initializing [ 4078.076055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4078.104271] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4082.613494] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4091.484578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4091.556502] Lustre: 172566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4091.750445] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4095.893198] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4104.398795] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4110.343607] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4116.772957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4117.201734] Lustre: lustre-OST0000: new disk, initializing [ 4117.208868] Lustre: Skipped 1 previous similar message [ 4117.213893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4117.221252] Lustre: Skipped 2 previous similar messages [ 4117.230118] Lustre: 174541:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4122.842456] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4125.368556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4125.384754] Lustre: Skipped 1 previous similar message [ 4125.395053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4125.662325] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4131.553961] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4141.035281] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4141.035589] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.037787] LustreError: Skipped 1 previous similar message [ 4141.044809] Lustre: Skipped 1 previous similar message [ 4141.050459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4141.053069] Lustre: Skipped 1 previous similar message [ 4145.776278] Lustre: server umount lustre-OST0000 complete [ 4151.275013] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4156.182634] Lustre: server umount lustre-MDT0000 complete [ 4156.388222] LustreError: 172573:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4156.426968] LustreError: 172573:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4161.506176] LustreError: 175213:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.535083] LustreError: 175213:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4161.804924] LustreError: 172558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788817900 with bad export cookie 4866665977674857562 [ 4161.809907] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4161.831850] LustreError: 172558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4162.568856] Lustre: server umount lustre-MDT0001 complete [ 4171.349658] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 17:51:48 (1788817908) [ 4188.274492] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 4200.483156] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 4241.757302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.186709] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4242.236262] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4242.430563] Lustre: lustre-MDT0000: new disk, initializing [ 4242.634074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4242.636883] Lustre: Skipped 2 previous similar messages [ 4242.678292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4247.290426] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4258.300938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4258.445457] Lustre: 178865:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4258.804695] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4263.491793] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4272.750547] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4278.337839] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4285.143663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4285.531135] Lustre: 180844:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4291.610638] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4292.707128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4292.934456] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4300.382269] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4308.462994] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.469905] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4308.509737] Lustre: Skipped 1 previous similar message [ 4308.525385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4308.542062] LustreError: Skipped 1 previous similar message [ 4308.585882] Lustre: Skipped 3 previous similar messages [ 4321.248654] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4321.375711] Lustre: server umount lustre-OST0000 complete [ 4325.351274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4331.780495] Lustre: server umount lustre-MDT0000 complete [ 4334.049441] LustreError: 178873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.091865] LustreError: 178873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4336.071923] LustreError: 178859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788818074 with bad export cookie 4866665977674858647 [ 4336.080134] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4336.085333] LustreError: 178859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4336.725194] Lustre: server umount lustre-MDT0001 complete [ 4347.411559] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 4350.025812] Key type lgssc unregistered [ 4350.328204] LNet: 182699:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4350.337306] LNetError: 182699:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4350.372464] LNet: Removed LNI 192.168.201.128@tcp [ 4351.450093] Key type .llcrypt unregistered [ 4351.451637] Key type ._llcrypt unregistered [ 4373.863441] Key type ._llcrypt registered [ 4373.865516] Key type .llcrypt registered [ 4373.955479] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 4375.358313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4375.445577] alg: No test for adler32 (adler32-zlib) [ 4376.599957] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 4376.912503] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 4378.616223] Key type lgssc registered [ 4379.658899] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4387.932393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4387.949580] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4389.599518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4393.886670] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4400.101073] LustreError: 184112:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4400.135776] LustreError: 184112:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4402.870698] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4403.306753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4407.549460] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4414.703303] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4421.286305] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4429.330459] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4429.832418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4435.415419] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4435.964691] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4443.568877] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4466.659774] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4466.664526] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4466.673388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4468.714509] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4468.722827] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4468.722867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4472.351668] Lustre: server umount lustre-OST0000 complete [ 4478.952656] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4478.965601] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4478.973623] Lustre: Skipped 1 previous similar message [ 4478.989824] Lustre: Skipped 2 previous similar messages [ 4482.482163] Lustre: server umount lustre-MDT0000 complete [ 4484.068393] LustreError: 184111:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4484.105270] LustreError: 184111:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4486.021359] LustreError: 184092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788818224 with bad export cookie 12868324350059950076 [ 4486.043846] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4486.361245] Lustre: server umount lustre-MDT0001 complete [ 4498.503265] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 4502.135300] Key type lgssc unregistered [ 4502.385863] LNet: 187666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4502.389192] LNetError: 187666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4502.402561] LNet: Removed LNI 192.168.201.128@tcp [ 4503.188981] Key type .llcrypt unregistered [ 4503.191284] Key type ._llcrypt unregistered [ 4528.306615] Key type ._llcrypt registered [ 4528.309695] Key type .llcrypt registered [ 4528.485661] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 4530.225665] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4530.318148] alg: No test for adler32 (adler32-zlib) [ 4531.460737] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 4531.806924] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 4533.552232] Key type lgssc registered [ 4535.154052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4543.395383] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4543.427270] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4544.843656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4549.401625] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4554.218485] LustreError: 189083:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4554.248240] LustreError: 189083:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4559.339041] LustreError: 189082:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4559.653080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4560.275811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4565.744356] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4572.909468] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4578.448910] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4586.036655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4586.389959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4589.517964] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4593.356383] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4602.601378] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4614.493911] Lustre: lustre-OST0000: Not available for connect from 192.168.201.28@tcp (stopping) [ 4615.156485] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4615.181185] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4615.204709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4616.172181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4616.177840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4618.333762] Lustre: server umount lustre-OST0000 complete [ 4626.840160] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8f9390031000.ost_server_uuid 50 [ 4628.995751] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4646.886034] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4646.928038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4646.942428] Lustre: Skipped 3 previous similar messages [ 4649.588071] Lustre: server umount lustre-MDT0000 complete [ 4652.001350] LustreError: 189077:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4652.017821] LustreError: 189077:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4653.381764] LustreError: 189063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788818392 with bad export cookie 3974984713752461653 [ 4653.395705] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4653.402866] LustreError: 189063:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4654.072802] Lustre: server umount lustre-MDT0001 complete [ 4662.259941] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 17:59:59 (1788818399) [ 4670.019904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4670.603940] LustreError: 192996:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4670.701784] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4676.077746] LustreError: 192997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4676.183363] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4685.060335] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4685.300574] LustreError: 192996:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.328663] LustreError: 192996:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4685.566845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4689.688676] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4696.161582] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4701.691988] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4709.983254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4710.376283] Lustre: lustre-OST0000: Not available for connect from 0@lo (not set up) [ 4710.523626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4715.637409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4717.104636] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4725.397849] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4733.091691] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4733.344679] Lustre: lustre-OST0001: new disk, initializing [ 4733.348188] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4733.356158] Lustre: 195849:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4733.485366] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4740.161974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4740.179248] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4740.296354] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4742.496884] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4753.738783] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4756.962810] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4756.975555] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4756.997469] Lustre: Skipped 1 previous similar message [ 4757.020981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4757.034444] Lustre: Skipped 1 previous similar message [ 4763.052810] Lustre: server umount lustre-OST0000 complete [ 4764.989263] LustreError: 194818:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4770.571209] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4770.910685] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4778.001867] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4778.285465] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4796.914921] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4796.915549] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4796.925364] LustreError: Skipped 1 previous similar message [ 4796.931466] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4796.951594] Lustre: Skipped 2 previous similar messages [ 4796.974776] Lustre: Skipped 4 previous similar messages [ 4798.836159] Lustre: server umount lustre-OST0001 complete [ 4803.557146] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4803.571664] LustreError: Skipped 1 previous similar message [ 4807.137291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4807.162203] Lustre: Skipped 1 previous similar message [ 4809.926931] Lustre: server umount lustre-MDT0000 complete [ 4812.258267] LustreError: 192997:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4812.264753] LustreError: 192997:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [ 4813.061095] LustreError: 192978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788818551 with bad export cookie 3974984713752462724 [ 4813.068058] LustreError: 192978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4813.071088] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4813.502666] Lustre: server umount lustre-MDT0001 complete [ 4837.041981] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:02:54 (1788818574) [ 4844.500498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4844.828191] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4844.865529] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4845.122227] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4850.277198] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4858.836715] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4858.964647] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4859.069635] Lustre: 199291:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4864.906428] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4872.011475] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4876.928702] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4884.063329] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4884.376485] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4890.870524] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4891.294512] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 4898.158926] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4905.198579] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4905.303890] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4913.061130] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 4913.866740] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 4922.513674] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4926.958240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4926.969847] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4926.986919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4926.989868] Lustre: Skipped 3 previous similar messages [ 4932.033601] Lustre: server umount lustre-OST0000 complete [ 4933.947776] LustreError: 201143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4949.985088] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4949.992388] LustreError: Skipped 2 previous similar messages [ 4953.917486] Lustre: server umount lustre-OST0001 complete [ 4960.231277] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4960.249646] Lustre: Skipped 3 previous similar messages [ 4960.259544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4960.265814] Lustre: Skipped 7 previous similar messages [ 4963.933573] Lustre: server umount lustre-MDT0000 complete [ 4967.546469] LustreError: 199283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788818706 with bad export cookie 3974984713752464131 [ 4967.560021] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4967.566257] LustreError: 199283:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4967.922671] Lustre: server umount lustre-MDT0001 complete [ 4992.748639] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:05:30 (1788818730) [ 5010.127596] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 5021.414707] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 5061.200757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5061.365720] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5061.414416] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5061.500223] Lustre: lustre-MDT0000: new disk, initializing [ 5061.574270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5061.579283] Lustre: Skipped 3 previous similar messages [ 5061.591259] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5065.304438] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5074.993585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5075.097145] Lustre: 207387:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5075.139216] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5075.142465] Lustre: Skipped 1 previous similar message [ 5075.192830] Lustre: lustre-MDT0001: new disk, initializing [ 5075.293281] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5075.311182] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5079.770666] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5089.185115] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5094.661658] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5101.228021] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5101.471037] Lustre: lustre-OST0000: new disk, initializing [ 5101.472906] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5101.480232] Lustre: 209350:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5107.386901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5107.404484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5107.570259] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5108.318674] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5119.351352] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5124.437557] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5124.630957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5129.346984] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5129.606197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5133.282695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5133.284061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5133.292263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5133.331941] Lustre: Skipped 2 previous similar messages [ 5137.860518] Lustre: server umount lustre-OST0000 complete [ 5156.320228] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5156.616655] Lustre: server umount lustre-MDT0000 complete [ 5158.885448] LustreError: 207396:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5158.894968] LustreError: 207396:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 5160.418822] LustreError: 207379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788818899 with bad export cookie 3974984713752465916 [ 5160.432890] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5160.733032] Lustre: server umount lustre-MDT0001 complete [ 5168.953605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5173.791304] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5182.071144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5186.394035] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5192.950665] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5198.086970] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5204.329290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5204.563396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5204.569039] Lustre: Skipped 4 previous similar messages [ 5210.215690] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5217.551593] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5221.448138] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5221.580305] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5225.477268] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5225.686668] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5228.514829] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5228.529929] LustreError: Skipped 1 previous similar message [ 5233.952899] Lustre: server umount lustre-OST0000 complete [ 5241.964192] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5247.591443] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5276.170350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5280.585634] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5280.612209] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5282.665504] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5289.491239] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5295.087978] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 4 sec [ 5295.978294] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5296.006052] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5299.824964] Lustre: server umount lustre-OST0000 complete [ 5305.313289] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5305.320552] Lustre: Skipped 4 previous similar messages [ 5305.327257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5305.332035] Lustre: Skipped 11 previous similar messages [ 5313.594840] Lustre: server umount lustre-MDT0000 complete [ 5317.423580] LustreError: 211551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788819056 with bad export cookie 3974984713752466798 [ 5317.435286] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5317.449326] LustreError: 211551:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5328.093973] Lustre: server umount lustre-MDT0001 complete [ 5337.723451] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:11:14 (1788819074) [ 5344.616711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5350.908076] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5358.497917] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.643270] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5370.591515] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5375.477859] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5381.437668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5387.345258] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5394.618965] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5400.055409] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5400.462114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5406.026237] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5406.380954] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5413.176498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5413.282598] Lustre: lustre-OST0001: new disk, initializing [ 5413.285686] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5413.289410] Lustre: 220906:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5418.573749] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5419.086033] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5419.101191] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5419.255863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5426.634701] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5430.857048] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5431.028375] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5435.443218] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5435.666496] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5439.976130] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5439.982388] LustreError: Skipped 3 previous similar messages [ 5443.586516] Lustre: server umount lustre-OST0001 complete [ 5445.096785] LustreError: 219359:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5445.117974] LustreError: 219359:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 9 previous similar messages [ 5449.281402] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5449.416386] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5454.285791] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5454.423457] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5484.318650] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5484.530788] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5484.548370] Lustre: Skipped 5 previous similar messages [ 5489.643979] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5489.656274] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5489.677593] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5489.681581] Lustre: Skipped 1 previous similar message [ 5490.675283] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5498.242305] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5499.724408] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5507.448666] Lustre: server umount lustre-OST0001 complete [ 5531.832756] Lustre: server umount lustre-OST0000 complete [ 5542.612497] Lustre: server umount lustre-MDT0000 complete [ 5546.908970] LustreError: 217518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788819285 with bad export cookie 3974984713752467764 [ 5546.913029] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5546.922164] LustreError: 217518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5561.312269] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5561.932469] Lustre: server umount lustre-MDT0001 complete [ 5589.309582] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:15:26 (1788819326) [ 5596.617067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5597.033354] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5597.087405] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5603.148415] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5614.588727] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5614.782427] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5614.845513] Lustre: 226154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5618.892799] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5627.157079] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5633.557210] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5642.743236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5643.181991] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5651.513378] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5659.104486] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5667.141098] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5667.219673] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5674.605357] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5683.508757] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5686.082715] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5692.454398] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5692.811413] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5699.052057] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5699.373104] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5707.022609] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8f9391ce6000.ost_server_uuid 50 [ 5709.931701] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8f9391ce6000.ost_server_uuid in IDLE state after 0 sec [ 5711.065346] Lustre: Permanently deactivating lustre-OST0001 [ 5711.109336] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5716.291476] Lustre: Permanently reactivating lustre-OST0001 [ 5716.303485] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 5716.314191] Lustre: Skipped 2 previous similar messages [ 5722.093984] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 5722.117223] LustreError: Skipped 5 previous similar messages [ 5722.133236] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.144241] Lustre: Skipped 9 previous similar messages [ 5722.151709] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5722.163410] Lustre: Skipped 17 previous similar messages [ 5726.955645] Lustre: server umount lustre-OST0001 complete [ 5746.144325] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5746.535598] Lustre: server umount lustre-OST0000 complete [ 5757.727698] Lustre: server umount lustre-MDT0000 complete [ 5763.174186] LustreError: 226146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788819502 with bad export cookie 3974984713752469052 [ 5763.176121] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5763.188744] LustreError: 226146:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5763.525808] Lustre: server umount lustre-MDT0001 complete [ 5796.560098] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:18:53 (1788819533) [ 5804.748870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5805.272894] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5805.348689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5805.370267] Lustre: Skipped 2 previous similar messages [ 5811.255580] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5822.467680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5822.522977] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5822.539903] Lustre: 232851:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5827.455217] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5837.688693] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5844.053957] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5852.050076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5852.236455] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5852.259773] Lustre: Permanently deactivating lustre-OST0000 [ 5852.273612] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5859.052827] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5860.396869] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5867.751495] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5874.522248] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5874.662907] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5881.086632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5881.470955] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 5889.736811] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5891.851175] Lustre: Permanently reactivating lustre-OST0000 [ 5891.861255] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5891.865432] Lustre: Skipped 2 previous similar messages [ 5897.979639] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5898.009660] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 37 seconds. I think it's dead, and I am evicting it. exp 0000000081d995f2, cur 1788819636 deadline 1788819629 last 1788819599 [ 5898.112704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5909.229857] Lustre: server umount lustre-OST0001 complete [ 5919.455238] Lustre: server umount lustre-OST0000 complete [ 5930.423377] Lustre: server umount lustre-MDT0000 complete [ 5934.970385] LustreError: 236451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788819673 with bad export cookie 3974984713752470718 [ 5934.976251] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5934.996870] LustreError: 236451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5935.742176] Lustre: server umount lustre-MDT0001 complete [ 5949.108741] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 5952.715629] Key type lgssc unregistered [ 5953.206613] LNet: 237735:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5953.217263] LNetError: 237735:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5953.228816] LNet: Removed LNI 192.168.201.128@tcp [ 5954.458380] Key type .llcrypt unregistered [ 5954.462812] Key type ._llcrypt unregistered [ 5981.576861] Key type ._llcrypt registered [ 5981.579911] Key type .llcrypt registered [ 5981.677965] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 5982.414258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5982.431566] alg: No test for adler32 (adler32-zlib) [ 5983.566592] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 5983.770112] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 5985.464846] Key type lgssc registered [ 5986.582628] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5996.406671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5996.429287] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5997.849961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6001.537461] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6007.269813] LustreError: 239200:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6007.299076] LustreError: 239200:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6010.856730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6010.939354] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6010.953370] Lustre: 239189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6010.965205] Lustre: Permanently deactivating lustre-MDT0001 [ 6010.980686] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6011.281204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6012.604753] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6012.624891] LustreError: 239231:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6012.647534] LustreError: 239231:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6012.673990] LustreError: 239231:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 6015.124748] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6022.501467] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6028.255290] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6037.955654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6038.364959] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6042.475043] LustreError: 241238:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6042.509221] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6045.154670] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6047.718755] LustreError: 241238:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.835885] LustreError: 241043:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6053.243610] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6056.948740] LustreError: 241238:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6056.974935] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6056.978705] LustreError: 241238:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6061.351676] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6061.565771] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6066.669977] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6066.692886] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6068.251215] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6076.688186] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6081.448411] Lustre: Permanently reactivating lustre-MDT0001 [ 6081.540747] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6081.543780] Lustre: Skipped 2 previous similar messages [ 6087.020549] Lustre: lustre-MDT0001: Client 501d6264-048b-4997-9c26-a3f32bf768a1 (at 192.168.201.28@tcp) reconnecting [ 6094.548683] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6096.372868] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6103.010647] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6103.365383] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6108.806070] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6109.106979] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6112.120816] Lustre: Permanently deactivating lustre-MDT0001 [ 6112.132348] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6112.136769] Lustre: Skipped 2 previous similar messages [ 6120.129711] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6128.100129] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6128.127836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6128.141889] Lustre: Skipped 3 previous similar messages [ 6129.294413] Lustre: server umount lustre-MDT0000 complete [ 6132.707107] LustreError: 239182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788819871 with bad export cookie 3415016447070353729 [ 6132.712712] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6132.721337] LustreError: 239182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6132.921577] Lustre: server umount lustre-MDT0001 complete [ 6142.971828] Lustre: server umount lustre-OST0000 complete [ 6155.746603] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:24:52 (1788819892) [ 6167.665559] Lustre: server umount lustre-OST0001 complete [ 6176.873269] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 6186.647299] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 6226.905569] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6227.163674] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6227.169837] Lustre: Skipped 2 previous similar messages [ 6227.189084] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6227.292173] Lustre: lustre-MDT0000: new disk, initializing [ 6227.408841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6227.445884] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6233.761112] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6247.705495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6247.849963] Lustre: 247461:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6247.936628] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6247.941218] Lustre: Skipped 1 previous similar message [ 6248.043382] Lustre: lustre-MDT0001: new disk, initializing [ 6248.138821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6248.169770] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6248.179570] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6253.207798] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6263.602784] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6270.855675] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6279.274535] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6279.556763] Lustre: lustre-OST0000: new disk, initializing [ 6279.560739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6279.567605] Lustre: 249430:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6279.685223] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6285.575312] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6287.978402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6287.995139] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6288.138792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6295.652757] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6297.989202] LustreError: 247468:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6306.675077] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6306.856763] Lustre: lustre-OST0001: new disk, initializing [ 6306.860506] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6306.866146] Lustre: 250546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6307.995869] LustreError: 247468:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6308.042345] LustreError: 247467:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6313.967384] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6316.620774] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6316.631952] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6316.739329] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6318.064142] LustreError: 247467:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6323.414947] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6325.230328] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6327.265336] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6327.268552] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6327.278059] LustreError: Skipped 1 previous similar message [ 6327.285214] Lustre: Skipped 3 previous similar messages [ 6327.287101] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6332.390466] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6332.398541] Lustre: Skipped 1 previous similar message [ 6333.391857] Lustre: server umount lustre-OST0001 complete [ 6337.507546] LustreError: 249414:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6337.547688] LustreError: 249414:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6352.887067] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6352.896565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6352.913613] Lustre: Skipped 2 previous similar messages [ 6352.932601] Lustre: Skipped 1 previous similar message [ 6356.717168] Lustre: server umount lustre-OST0000 complete [ 6361.058177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6361.081990] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6361.103505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6366.630168] Lustre: server umount lustre-MDT0000 complete [ 6368.232653] LustreError: 248643:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6368.258443] LustreError: 248643:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 6370.412671] LustreError: 247453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788820109 with bad export cookie 3415016447070356536 [ 6370.419872] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6370.423460] LustreError: 247453:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6376.852801] Lustre: server umount lustre-MDT0001 complete [ 6389.303851] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 6392.027625] Key type lgssc unregistered [ 6392.427735] LNet: 252579:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6392.439839] LNetError: 252579:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6393.510718] LNet: Removed LNI 192.168.201.128@tcp [ 6394.706721] Key type .llcrypt unregistered [ 6394.708235] Key type ._llcrypt unregistered [ 6439.566975] Key type ._llcrypt registered [ 6439.571224] Key type .llcrypt registered [ 6439.708698] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 6440.722455] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6440.759464] alg: No test for adler32 (adler32-zlib) [ 6441.916833] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 6442.191665] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 6443.848242] Key type lgssc registered [ 6445.063774] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6453.533977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6453.558968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6454.847712] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6454.862395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6455.000554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6460.370709] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6470.307754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6470.386310] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6470.423288] Lustre: 254786:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6470.830201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6477.128858] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6486.017498] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6492.062814] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6499.858232] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6500.209222] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6500.578569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6510.253513] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6510.846987] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6518.745351] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6575.301390] Lustre: server umount lustre-OST0000 complete [ 6581.343581] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6594.001314] Lustre: 254109:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788820316/real 1788820316] req@00000000879af25c x1875713910159104/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788820332 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6594.024627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6594.028191] Lustre: 254109:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6594.049359] Lustre: Skipped 1 previous similar message [ 6595.067369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6595.458462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6598.560315] Lustre: 254109:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788820321/real 1788820321] req@000000005cae140c x1875713910160768/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788820337 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6598.587859] Lustre: 254109:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 6602.037339] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6603.233097] Lustre: 254107:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788820326/real 1788820326] req@00000000da3b8c17 x1875713910161920/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788820342 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6603.636872] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6603.661511] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6603.689369] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6603.693026] Lustre: Skipped 1 previous similar message [ 6610.162907] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6649.832844] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6649.835772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6649.845408] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6649.880610] Lustre: Skipped 1 previous similar message [ 6654.950440] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6654.956119] Lustre: Skipped 1 previous similar message [ 6655.171651] Lustre: server umount lustre-OST0000 complete [ 6660.067810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6660.069818] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6660.080906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6660.080919] Lustre: Skipped 1 previous similar message [ 6660.087247] Lustre: Skipped 1 previous similar message [ 6660.125799] LustreError: Skipped 1 previous similar message [ 6665.188354] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6665.195689] Lustre: Skipped 2 previous similar messages [ 6670.308194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6673.376254] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6673.769559] Lustre: server umount lustre-MDT0000 complete [ 6677.419541] LustreError: 254780:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788820416 with bad export cookie 9534849991453924081 [ 6677.436401] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6677.743334] Lustre: server umount lustre-MDT0001 complete [ 6688.909145] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 6691.909172] Key type lgssc unregistered [ 6692.183890] LNet: 260206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6692.198664] LNetError: 260206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6692.217937] LNet: Removed LNI 192.168.201.128@tcp [ 6693.269347] Key type .llcrypt unregistered [ 6693.271694] Key type ._llcrypt unregistered [ 6717.199421] Key type ._llcrypt registered [ 6717.201752] Key type .llcrypt registered [ 6717.318864] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 6718.916862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6718.988121] alg: No test for adler32 (adler32-zlib) [ 6720.124218] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 6720.423798] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 6722.210085] Key type lgssc registered [ 6723.421919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6731.209065] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6731.228762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6732.877697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6737.994729] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6743.016448] LustreError: 261620:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6743.046918] LustreError: 261620:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6746.250726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6746.597338] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6751.812944] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6759.797241] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6766.313216] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6772.998127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6773.487799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6777.598406] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6781.764727] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6791.013392] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6817.762875] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6817.771884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6817.772097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6817.774062] Lustre: Skipped 1 previous similar message [ 6822.379617] Lustre: server umount lustre-OST0000 complete [ 6828.004841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6828.014642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6828.021376] Lustre: Skipped 2 previous similar messages [ 6831.010198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6831.017088] Lustre: Skipped 1 previous similar message [ 6832.808303] Lustre: server umount lustre-MDT0000 complete [ 6836.197223] LustreError: 262348:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6836.224508] LustreError: 262348:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6836.612834] LustreError: 262356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788820575 with bad export cookie 1194752554985574380 [ 6836.617693] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6836.627962] LustreError: 262356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6836.998664] Lustre: server umount lustre-MDT0001 complete [ 6849.891349] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 6853.409517] Key type lgssc unregistered [ 6853.758846] LNet: 265929:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6853.768820] LNetError: 265929:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6853.787955] LNet: Removed LNI 192.168.201.128@tcp [ 6854.782634] Key type .llcrypt unregistered [ 6854.787108] Key type ._llcrypt unregistered [ 6873.224591] Key type ._llcrypt registered [ 6873.230625] Key type .llcrypt registered [ 6873.367643] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 6874.692434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6874.829165] alg: No test for adler32 (adler32-zlib) [ 6876.021708] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 6876.278703] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 6877.872191] Key type lgssc registered [ 6879.626669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6889.723877] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6889.739201] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6891.279523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6895.424075] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6901.737155] LustreError: 267152:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6901.760313] LustreError: 267152:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6904.800528] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6905.459305] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6910.679630] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6920.791437] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6927.551453] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6936.481422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6937.123978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6942.201924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6944.396155] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 6951.027615] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6962.657616] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6962.664413] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6962.681905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6964.207686] Lustre: server umount lustre-OST0000 complete [ 6970.861872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6970.888433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6970.891347] Lustre: Skipped 1 previous similar message [ 6970.907637] Lustre: Skipped 1 previous similar message [ 6974.911674] Lustre: server umount lustre-MDT0000 complete [ 6975.981758] LustreError: 267152:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6976.008766] LustreError: 267152:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6978.694815] LustreError: 269874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788820717 with bad export cookie 1114956885779727569 [ 6978.699257] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6978.712876] LustreError: 269874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6988.458195] Lustre: server umount lustre-MDT0001 complete [ 6998.587629] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 7001.479541] Key type lgssc unregistered [ 7001.712645] LNet: 270826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7001.720183] LNetError: 270826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7001.752499] LNet: Removed LNI 192.168.201.128@tcp [ 7002.803390] Key type .llcrypt unregistered [ 7002.807165] Key type ._llcrypt unregistered [ 7025.571836] Key type ._llcrypt registered [ 7025.574091] Key type .llcrypt registered [ 7025.656253] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 7026.444807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7026.466314] alg: No test for adler32 (adler32-zlib) [ 7027.547335] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 7027.774679] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 7029.488228] Key type lgssc registered [ 7030.621141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7036.996627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7037.008224] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7038.363794] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7042.479733] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7048.685445] LustreError: 272236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7048.721407] LustreError: 272236:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7050.585146] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7050.960830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7055.604937] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7062.880604] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7068.908958] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7074.756632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7075.107059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7080.577380] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7083.317581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7088.520371] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7117.282804] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7117.287393] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7117.296184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7119.334784] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7119.343428] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7119.356334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7119.382351] Lustre: Skipped 1 previous similar message [ 7121.372203] Lustre: server umount lustre-OST0000 complete [ 7128.033238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7128.043913] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7128.060080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7134.700812] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7134.708651] Lustre: Skipped 3 previous similar messages [ 7139.296922] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7139.790552] Lustre: server umount lustre-MDT0000 complete [ 7139.820304] LustreError: 272236:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7139.835307] LustreError: 272236:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7144.426838] LustreError: 272218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788820883 with bad export cookie 8609165871716314890 [ 7144.430745] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7144.435662] LustreError: 272218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7145.042865] Lustre: server umount lustre-MDT0001 complete [ 7157.913239] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 7161.143349] Key type lgssc unregistered [ 7161.466263] LNet: 276546:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7161.477960] LNetError: 276546:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7161.493758] LNet: Removed LNI 192.168.201.128@tcp [ 7162.779331] Key type .llcrypt unregistered [ 7162.783300] Key type ._llcrypt unregistered [ 7182.831321] Key type ._llcrypt registered [ 7182.835316] Key type .llcrypt registered [ 7182.939288] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 7184.312152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7184.332995] alg: No test for adler32 (adler32-zlib) [ 7185.335280] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 7185.617559] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 7187.328482] Key type lgssc registered [ 7188.700258] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7196.887206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7196.920072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7198.475929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7203.215804] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7208.936549] LustreError: 277779:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7208.964970] LustreError: 277779:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7212.815893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7213.292524] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7217.522110] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7224.507414] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7230.974671] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7239.083334] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7239.701511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7243.864330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7246.048273] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7251.449940] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7273.447844] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7273.448639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7273.456127] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7273.459307] Lustre: Skipped 1 previous similar message [ 7273.467428] Lustre: Skipped 1 previous similar message [ 7278.290691] Lustre: server umount lustre-OST0000 complete [ 7283.682532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7283.684214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7283.690886] Lustre: Skipped 1 previous similar message [ 7287.557352] Lustre: server umount lustre-MDT0000 complete [ 7288.805708] LustreError: 277790:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7288.817255] LustreError: 277790:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7290.155143] LustreError: 277745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788821029 with bad export cookie 16396106524871993047 [ 7290.156431] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7290.169893] LustreError: 277745:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7290.554275] Lustre: server umount lustre-MDT0001 complete [ 7301.366828] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 7304.300841] Key type lgssc unregistered [ 7304.836671] LNet: 281500:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7304.862261] LNetError: 281500:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7304.897559] LNet: Removed LNI 192.168.201.128@tcp [ 7305.892139] Key type .llcrypt unregistered [ 7305.894505] Key type ._llcrypt unregistered [ 7319.213193] Key type ._llcrypt registered [ 7319.214875] Key type .llcrypt registered [ 7319.344086] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7340.504533] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 7355.132298] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 7357.374405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7357.497180] alg: No test for adler32 (adler32-zlib) [ 7358.896771] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 7359.383542] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 7361.208205] Key type lgssc registered [ 7362.588669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7403.034040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7403.107649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7404.445849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7404.498647] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7404.653261] Lustre: lustre-MDT0000: new disk, initializing [ 7404.725307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7404.739047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7409.774543] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7422.872878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7423.046255] Lustre: 285051:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7423.232856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7423.240147] Lustre: Skipped 1 previous similar message [ 7423.318477] Lustre: lustre-MDT0001: new disk, initializing [ 7423.404646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7423.440789] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7423.452500] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7428.839398] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7438.717622] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7444.870454] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7450.702210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7450.926369] Lustre: lustre-OST0000: new disk, initializing [ 7450.929697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7450.936101] Lustre: 287025:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7451.028541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7456.400147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7456.406525] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7456.546575] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7456.577782] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7465.874667] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7471.633567] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7471.960822] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7477.392162] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7477.641971] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7482.338660] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7482.341963] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7482.346196] LustreError: Skipped 1 previous similar message [ 7482.353027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7482.359731] Lustre: Skipped 1 previous similar message [ 7482.376167] Lustre: Skipped 1 previous similar message [ 7485.520771] Lustre: server umount lustre-OST0000 complete [ 7490.028871] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7490.034299] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7490.051500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7492.577761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7492.578100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7494.781165] Lustre: server umount lustre-MDT0000 complete [ 7497.721238] LustreError: 285848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788821236 with bad export cookie 7845853584839717745 [ 7497.728122] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7497.733202] LustreError: 285848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7498.002403] Lustre: server umount lustre-MDT0001 complete [ 7505.516452] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 18:47:23 (1788821243) [ 7510.816464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7511.286281] LustreError: 289436:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7511.303215] LustreError: 289436:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7511.350575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7515.968558] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7516.641847] LustreError: 289437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7521.763464] LustreError: 289436:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7523.267048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7523.678058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7528.253436] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7534.769897] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7540.649923] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7548.437305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7548.670741] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7554.657426] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7563.087242] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7578.592904] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7578.595673] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7578.601145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7578.603875] Lustre: Skipped 1 previous similar message [ 7582.661426] Lustre: server umount lustre-OST0000 complete [ 7589.857610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7589.862499] Lustre: Skipped 1 previous similar message [ 7589.865216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7589.867662] Lustre: Skipped 3 previous similar messages [ 7593.672519] Lustre: server umount lustre-MDT0000 complete [ 7594.979937] LustreError: 289437:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7594.997918] LustreError: 289437:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7597.994944] LustreError: 289418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788821336 with bad export cookie 7845853584839718627 [ 7598.003417] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7598.005665] LustreError: 289418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7598.508715] Lustre: server umount lustre-MDT0001 complete [ 7609.269711] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 7611.305662] Key type lgssc unregistered [ 7611.559652] LNet: 292991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7611.567914] LNetError: 292991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7611.580533] LNet: Removed LNI 192.168.201.128@tcp [ 7612.465340] Key type .llcrypt unregistered [ 7612.467339] Key type ._llcrypt unregistered [ 7641.076787] Key type ._llcrypt registered [ 7641.079292] Key type .llcrypt registered [ 7641.185395] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 7657.753457] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 7658.694269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7658.721375] alg: No test for adler32 (adler32-zlib) [ 7659.810937] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 7660.036413] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 7661.744297] Key type lgssc registered [ 7663.263064] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7711.868334] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7711.906729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7713.194545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7713.232886] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7713.333972] Lustre: lustre-MDT0000: new disk, initializing [ 7713.437350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.476762] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7718.459969] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7728.675897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7728.763400] Lustre: 296817:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7728.810724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7728.815547] Lustre: Skipped 1 previous similar message [ 7728.880622] Lustre: lustre-MDT0001: new disk, initializing [ 7728.966623] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7729.002176] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7729.017966] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7733.669181] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7742.999502] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7748.174532] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7754.556794] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7754.901775] Lustre: lustre-OST2710: new disk, initializing [ 7754.907876] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7754.913912] Lustre: 298786:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7754.994984] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7760.977919] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7764.085295] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7764.102194] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7764.201076] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7771.171896] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7778.898709] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7779.074223] Lustre: lustre-OST03e8: new disk, initializing [ 7779.076824] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7779.081891] Lustre: 299850:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7779.153341] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7786.373947] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7787.217171] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7787.238186] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7787.383245] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7793.961750] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7800.072723] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7800.292748] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7805.611910] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7805.936472] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7810.911696] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7811.254500] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7818.022258] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7818.318152] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7823.329992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7823.337313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7823.337368] Lustre: Skipped 1 previous similar message [ 7823.356263] Lustre: Skipped 3 previous similar messages [ 7825.360792] Lustre: server umount lustre-MDT0000 complete [ 7830.146490] LustreError: 296809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788821569 with bad export cookie 1296520000901712359 [ 7830.191804] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7830.204196] LustreError: 296809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7830.796228] Lustre: server umount lustre-MDT0001 complete [ 7844.390208] Lustre: server umount lustre-OST2710 complete [ 7859.272170] Lustre: server umount lustre-OST03e8 complete [ 7879.859985] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 7889.953386] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 7928.263375] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.573656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7928.612808] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7928.750203] Lustre: lustre-MDT0000: new disk, initializing [ 7928.848491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7928.883570] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7934.013251] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7945.302826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7945.429777] Lustre: 305102:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7945.482945] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7945.486584] Lustre: Skipped 1 previous similar message [ 7945.548195] Lustre: lustre-MDT0001: new disk, initializing [ 7945.630311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7945.659571] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7945.671980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7949.819556] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7959.413333] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7964.891918] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7971.451514] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7971.728494] Lustre: lustre-OST0000: new disk, initializing [ 7971.737959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7971.743461] Lustre: 307071:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7971.844858] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7977.750888] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 7978.062498] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7978.081860] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7978.169941] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7987.247770] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7992.729608] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7992.991891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7996.968536] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7997.223197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8004.065830] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8004.082789] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8004.107940] Lustre: Skipped 2 previous similar messages [ 8004.112924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8004.128480] Lustre: Skipped 1 previous similar message [ 8009.193560] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8009.204362] Lustre: Skipped 1 previous similar message [ 8013.281839] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8013.530648] Lustre: server umount lustre-OST0000 complete [ 8019.427761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8019.452202] Lustre: Skipped 1 previous similar message [ 8019.461263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8019.469435] Lustre: Skipped 1 previous similar message [ 8023.810539] Lustre: server umount lustre-MDT0000 complete [ 8024.545616] LustreError: 306973:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8024.556611] LustreError: 306973:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 8027.109520] LustreError: 305092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788821766 with bad export cookie 1296520000901714039 [ 8027.116060] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8027.122525] LustreError: 305092:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8027.585683] Lustre: server umount lustre-MDT0001 complete [ 8035.565932] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 18:56:12 (1788821772) [ 8036.955393] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8039.463408] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 18:56:15 (1788821775) [ 8043.004165] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 8072.462247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8072.698957] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8072.763066] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8072.971893] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8077.113952] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8083.599702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8083.697240] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8083.726441] Lustre: 310929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8088.160983] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8109.682509] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8109.935650] Lustre: Denying initial registration attempt from nid 192.168.201.128@tcp, specified as failover [ 8109.946748] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8109.956750] LustreError: 312383:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [ 8109.962704] LustreError: 312383:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8109.972246] LustreError: 312383:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8110.070293] Lustre: server umount lustre-OST0000 complete [ 8110.072996] LustreError: 312383:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8119.779623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8119.786229] LustreError: Skipped 1 previous similar message [ 8119.796413] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8119.811532] Lustre: Skipped 1 previous similar message [ 8119.827375] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8123.386230] Lustre: server umount lustre-MDT0000 complete [ 8125.411646] LustreError: 310941:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8125.430215] LustreError: 310941:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8127.209932] LustreError: 310920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788821866 with bad export cookie 1296520000901714921 [ 8127.215132] LustreError: 310920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8127.215648] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8127.414253] Lustre: server umount lustre-MDT0001 complete [ 8137.637898] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 8141.967887] Key type lgssc unregistered [ 8142.259880] LNet: 313502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8142.272782] LNetError: 313502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8142.285565] LNet: Removed LNI 192.168.201.128@tcp [ 8143.130707] Key type .llcrypt unregistered [ 8143.135119] Key type ._llcrypt unregistered [ 8157.002212] Key type ._llcrypt registered [ 8157.004665] Key type .llcrypt registered [ 8157.147101] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 8158.160076] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8158.265832] alg: No test for adler32 (adler32-zlib) [ 8159.364488] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 8159.640768] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 8161.328205] Key type lgssc registered [ 8162.501679] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8210.963355] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 8219.827356] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8219.844370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8221.045601] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8221.066783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8221.268324] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8225.494893] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8233.519102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8233.618199] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8233.640831] Lustre: 316263:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8233.841653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8237.314435] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8246.660582] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8254.834954] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8277.423985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8277.731855] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8277.887469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8283.892088] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8294.936447] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8302.568980] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8302.571453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8302.584180] LustreError: Skipped 1 previous similar message [ 8302.585219] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8302.628125] Lustre: Skipped 1 previous similar message [ 8304.996130] Lustre: server umount lustre-OST0000 complete [ 8310.754987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8310.773619] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8310.804454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8310.817450] Lustre: Skipped 1 previous similar message [ 8312.804042] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8312.806401] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8316.673412] Lustre: server umount lustre-MDT0000 complete [ 8320.465926] LustreError: 317004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788822059 with bad export cookie 12418562595552506580 [ 8320.499499] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8320.826953] Lustre: server umount lustre-MDT0001 complete [ 8332.917833] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 8335.769210] Key type lgssc unregistered [ 8336.163084] LNet: 319847:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8336.169789] LNetError: 319847:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8336.203741] LNet: Removed LNI 192.168.201.128@tcp [ 8337.235444] Key type .llcrypt unregistered [ 8337.238171] Key type ._llcrypt unregistered [ 8361.608791] Key type ._llcrypt registered [ 8361.611416] Key type .llcrypt registered [ 8361.756961] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 8363.233912] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8363.272067] alg: No test for adler32 (adler32-zlib) [ 8364.378922] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 8364.602097] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 8366.304275] Key type lgssc registered [ 8367.469777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8375.224677] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8375.234360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8376.829148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8381.599816] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8387.043266] LustreError: 321260:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8387.049793] LustreError: 321260:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8389.811896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8390.210723] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8394.192822] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8400.631633] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8406.814618] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8416.646079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8417.302477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8424.512458] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8431.820961] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8445.920805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8445.943088] Lustre: Skipped 2 previous similar messages [ 8445.951203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8445.960147] Lustre: Skipped 2 previous similar messages [ 8449.857923] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.28@tcp (stopping) [ 8451.056421] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8451.072836] Lustre: Skipped 2 previous similar messages [ 8451.370273] Lustre: server umount lustre-MDT0000 complete [ 8454.766632] LustreError: 321996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788822193 with bad export cookie 13210821839465719137 [ 8454.770958] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8454.783758] LustreError: 321996:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8454.970325] LustreError: 321256:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8455.012537] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.28@tcp (stopping) [ 8456.161929] LustreError: 321255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8456.163391] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8459.169983] LustreError: 321255:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8459.171472] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8459.186731] Lustre: Skipped 1 previous similar message [ 8461.226361] Lustre: server umount lustre-MDT0001 complete [ 8467.321892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8475.273148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8479.715184] LustreError: 320568:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001429fa75 x1875715925512704/t0(0) o250->MGC192.168.201.128@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8480.058300] LustreError: 323094:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8480.075954] LustreError: 323094:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8480.165748] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8480.223637] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8484.586305] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8489.382256] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8489.387226] LustreError: 323095:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8489.406231] LustreError: 323095:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8493.050028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8493.540418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8498.452953] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8498.664249] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8506.133980] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8507.622583] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8512.642919] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8513.961597] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8519.146505] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8519.147425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8519.151323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8519.151333] Lustre: Skipped 1 previous similar message [ 8519.171949] Lustre: Skipped 1 previous similar message [ 8523.822813] Lustre: server umount lustre-OST0000 complete [ 8529.393392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8529.402741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8534.501652] LustreError: 324877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8534.522110] LustreError: 324877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8534.689806] Lustre: server umount lustre-MDT0000 complete [ 8539.013479] LustreError: 321239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788822277 with bad export cookie 13210821839465726543 [ 8539.018121] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8539.031492] LustreError: 321239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8539.724232] Lustre: server umount lustre-MDT0001 complete [ 8551.652907] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 8554.256446] Key type lgssc unregistered [ 8554.563221] LNet: 327543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8554.570664] LNetError: 327543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8554.585182] LNet: Removed LNI 192.168.201.128@tcp [ 8555.459262] Key type .llcrypt unregistered [ 8555.462131] Key type ._llcrypt unregistered [ 8578.131775] Key type ._llcrypt registered [ 8578.133466] Key type .llcrypt registered [ 8578.245255] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 8579.410620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8579.425556] alg: No test for adler32 (adler32-zlib) [ 8580.373219] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 8580.514579] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 8582.152627] Key type lgssc registered [ 8583.167456] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8589.520646] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8589.536699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8590.854884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8594.586071] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8601.061823] LustreError: 328957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8601.100225] LustreError: 328957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8601.287132] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.558912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8604.952240] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8610.566103] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8614.981091] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8621.031825] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8621.034188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8621.036343] Lustre: Skipped 1 previous similar message [ 8621.045417] Lustre: Skipped 1 previous similar message [ 8623.695830] Lustre: server umount lustre-MDT0000 complete [ 8626.146476] LustreError: 328957:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8626.164546] LustreError: 328957:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8627.247844] LustreError: 328937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788822366 with bad export cookie 17979651561096063097 [ 8627.249271] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8627.261196] LustreError: 328937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8627.638839] Lustre: server umount lustre-MDT0001 complete [ 8642.212849] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8642.458318] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8642.479181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8642.633917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8647.603717] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8657.400667] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8657.494288] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8657.534896] Lustre: 331544:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8657.853499] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8662.442083] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8670.348211] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8675.676717] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8685.377348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8685.595955] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8685.603773] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8685.617472] LustreError: 333425:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [ 8685.623184] LustreError: 333425:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [ 8685.628543] LustreError: 333425:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8685.711543] Lustre: server umount lustre-OST0000 complete [ 8685.714568] LustreError: 333425:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8695.980123] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8696.193084] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8696.423433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8702.359404] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8705.213824] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8709.525679] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8716.014989] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8716.113800] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8716.184267] Lustre: lustre-OST0001: new disk, initializing [ 8716.188333] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8716.193966] Lustre: 334967:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8716.283465] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8722.939821] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8724.581370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8724.586569] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8724.671601] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8730.912892] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8735.206970] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8735.211703] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8735.232284] Lustre: Skipped 1 previous similar message [ 8735.252114] Lustre: Skipped 1 previous similar message [ 8740.089222] Lustre: server umount lustre-OST0001 complete [ 8740.352538] LustreError: 335401:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8740.393402] LustreError: 335401:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8745.443369] LustreError: 333894:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8745.451920] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8745.457954] LustreError: 333894:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8745.483851] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8745.494480] Lustre: Skipped 1 previous similar message [ 8750.579076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8750.582718] LustreError: 333892:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8750.586571] Lustre: Skipped 1 previous similar message [ 8750.610677] LustreError: 333892:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8755.683533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8755.697657] Lustre: Skipped 1 previous similar message [ 8758.240145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8758.506768] Lustre: server umount lustre-OST0000 complete [ 8765.410598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8765.419496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8765.431254] Lustre: Skipped 1 previous similar message [ 8765.443189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8769.338502] Lustre: server umount lustre-MDT0000 complete [ 8771.051269] LustreError: 331556:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8771.087526] LustreError: 331556:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 8773.371234] LustreError: 331537:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788822512 with bad export cookie 17979651561096063650 [ 8773.377821] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8773.660730] Lustre: server umount lustre-MDT0001 complete [ 8786.887498] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 8789.975229] Key type lgssc unregistered [ 8790.239536] LNet: 336928:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8790.255257] LNetError: 336928:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8790.281921] LNet: Removed LNI 192.168.201.128@tcp [ 8791.370911] Key type .llcrypt unregistered [ 8791.375917] Key type ._llcrypt unregistered [ 8863.063277] Key type ._llcrypt registered [ 8863.068342] Key type .llcrypt registered [ 8863.195639] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 8879.591781] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 8881.413258] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8881.477902] alg: No test for adler32 (adler32-zlib) [ 8882.509908] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 8882.675341] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 8884.336232] Key type lgssc registered [ 8885.701898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8931.318921] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8931.337820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8932.705366] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8932.732546] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8932.826095] Lustre: lustre-MDT0000: new disk, initializing [ 8932.902940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8932.923427] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8937.418980] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8948.126389] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8948.286501] Lustre: 341860:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8948.370048] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8948.378775] Lustre: Skipped 1 previous similar message [ 8948.499398] Lustre: lustre-MDT0001: new disk, initializing [ 8948.608621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8948.641950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8948.663160] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8953.350647] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8964.633732] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8970.383578] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8977.256262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8977.608754] Lustre: lustre-OST0000: new disk, initializing [ 8977.612105] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8977.617883] Lustre: 343830:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8977.694424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8982.615642] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 8986.724305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8986.734404] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8986.859258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8991.661985] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8996.342833] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8996.475913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9001.397289] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9001.724057] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9007.590556] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9007.593409] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9007.600781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9007.620826] LustreError: Skipped 1 previous similar message [ 9009.857550] Lustre: server umount lustre-OST0000 complete [ 9015.267883] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9015.281242] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9015.307849] Lustre: Skipped 1 previous similar message [ 9015.312741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9015.319435] Lustre: Skipped 1 previous similar message [ 9017.829055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9017.829667] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9017.870536] Lustre: Skipped 1 previous similar message [ 9019.953957] Lustre: server umount lustre-MDT0000 complete [ 9024.148672] LustreError: 342665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788822763 with bad export cookie 2216764632229191726 [ 9024.152014] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9024.157753] LustreError: 342665:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9024.546533] Lustre: server umount lustre-MDT0001 complete [ 9033.389810] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:12:50 (1788822770) [ 9042.233218] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:12:58 (1788822778) [ 9053.105024] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9053.794026] LustreError: 346588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9053.827224] LustreError: 346588:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9053.917837] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9058.986932] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9059.304235] LustreError: 346589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9063.396205] LustreError: 346588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9066.912088] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9067.157072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9071.965602] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9078.491659] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9083.812069] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9090.119420] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9090.477697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9096.976348] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9104.015956] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9111.213152] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9112.745460] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9114.082068] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9115.592160] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9117.197365] Lustre: Failing over lustre-MDT0000 [ 9117.471960] Lustre: server umount lustre-MDT0000 complete [ 9118.694638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9118.697808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9118.725333] LustreError: 346588:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9124.329567] LustreError: 346584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9124.354282] LustreError: 346584:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9134.561278] LustreError: 346589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9134.577654] LustreError: 346589:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 9135.584920] Lustre: 340084:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788822858/real 1788822858] req@00000000b35f8a39 x1875716468706688/t0(0) o400->MGC192.168.201.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788822874 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9135.604117] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9135.914615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9145.825469] LustreError: 349782:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 9146.115674] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9146.209240] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9150.649451] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9151.286506] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9151.474071] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9151.509784] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9151.590243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9159.472330] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9160.699765] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9164.136856] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9165.918376] Lustre: Failing over lustre-MDT0000 [ 9166.371093] Lustre: server umount lustre-MDT0000 complete [ 9166.664701] LustreError: 350274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9166.675294] LustreError: 350274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 9166.817695] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9166.846433] Lustre: Skipped 4 previous similar messages [ 9182.176787] Lustre: 340081:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788822905/real 1788822905] req@00000000d68f27e1 x1875716468726784/t0(0) o400->MGC192.168.201.128@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788822921 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9182.201863] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9184.342553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9192.661170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.702382] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9193.269782] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9196.564400] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9198.081540] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9198.095973] Lustre: Skipped 2 previous similar messages [ 9198.132254] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9198.180543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9205.704968] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9207.318190] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9213.409501] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9213.412418] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9213.416147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9213.419839] Lustre: Skipped 1 previous similar message [ 9213.430153] Lustre: Skipped 2 previous similar messages [ 9217.995791] Lustre: server umount lustre-MDT0000 complete [ 9218.534696] LustreError: 346585:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9218.564214] LustreError: 346585:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 25 previous similar messages [ 9222.105129] LustreError: 346569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788822961 with bad export cookie 2216764632229194435 [ 9222.111698] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9222.113197] LustreError: 346569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9222.392413] Lustre: server umount lustre-MDT0001 complete [ 9236.113260] Lustre: server umount lustre-OST0000 complete [ 9251.527653] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9256.640081] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9265.246790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9265.648139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9265.651745] Lustre: Skipped 1 previous similar message [ 9270.330488] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9278.817506] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9285.932137] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9293.332417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9296.687567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9298.912200] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9305.119701] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9322.479077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9322.497337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9322.522500] Lustre: Skipped 1 previous similar message [ 9322.534530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9328.296849] Lustre: server umount lustre-OST0000 complete [ 9336.805313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9336.816753] Lustre: Skipped 3 previous similar messages [ 9338.693644] Lustre: server umount lustre-MDT0000 complete [ 9341.922649] LustreError: 352928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9341.951560] LustreError: 352928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9342.310661] LustreError: 352909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788823081 with bad export cookie 2216764632229195184 [ 9342.313719] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9342.316621] LustreError: 352909:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9342.575186] Lustre: server umount lustre-MDT0001 complete [ 9353.861032] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 9356.702885] Key type lgssc unregistered [ 9356.938577] LNet: 356481:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9356.944343] LNetError: 356481:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9357.997517] LNet: Removed LNI 192.168.201.128@tcp [ 9358.746747] Key type .llcrypt unregistered [ 9358.752024] Key type ._llcrypt unregistered [ 9392.761569] Key type ._llcrypt registered [ 9392.762657] Key type .llcrypt registered [ 9392.819156] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 9394.265177] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9394.273854] alg: No test for adler32 (adler32-zlib) [ 9395.376833] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 9395.686712] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 9397.409099] Key type lgssc registered [ 9399.146445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9408.542219] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9408.554282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9410.034695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9415.617710] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9419.240606] LustreError: 357995:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9419.260594] LustreError: 357995:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9424.365884] LustreError: 357996:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9425.202800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9425.710532] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9430.985368] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9438.467972] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9443.520078] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9450.462512] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9451.079723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9455.184871] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9458.899825] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9469.504140] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9480.677610] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9480.684086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9480.698591] Lustre: Skipped 2 previous similar messages [ 9480.701015] Lustre: Skipped 1 previous similar message [ 9485.794983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9485.806196] Lustre: Skipped 3 previous similar messages [ 9486.314923] Lustre: server umount lustre-MDT0000 complete [ 9489.812440] LustreError: 357978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788823228 with bad export cookie 17057990664250362743 [ 9489.813797] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9489.819955] LustreError: 357978:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9490.026622] Lustre: server umount lustre-MDT0001 complete [ 9504.322650] Lustre: server umount lustre-OST0000 complete [ 9518.713553] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9519.629402] LustreError: 361708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9519.791740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9524.951209] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9525.220099] LustreError: 361709:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9529.320505] LustreError: 361708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9533.226599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9533.697738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9536.807858] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9544.218976] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9550.083429] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9558.847373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9559.366671] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9564.481980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9568.517016] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9577.343850] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9589.219657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9589.223297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9589.234422] Lustre: Skipped 2 previous similar messages [ 9589.242178] Lustre: Skipped 1 previous similar message [ 9594.337610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9594.343794] Lustre: Skipped 3 previous similar messages [ 9594.514222] Lustre: server umount lustre-MDT0000 complete [ 9598.539542] LustreError: 361690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788823337 with bad export cookie 17057990664250363856 [ 9598.541702] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9598.561135] LustreError: 361690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9599.014347] Lustre: server umount lustre-MDT0001 complete [ 9613.941296] Lustre: server umount lustre-OST0000 complete [ 9635.183063] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9635.808074] LustreError: 365623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9635.948421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9642.217765] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9652.127224] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9658.270302] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9667.872171] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9674.358592] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9680.742508] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9681.135476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9681.143157] Lustre: Skipped 1 previous similar message [ 9689.583612] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9699.946146] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9705.727755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9711.073969] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9711.084068] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9711.091111] Lustre: Skipped 1 previous similar message [ 9711.110735] Lustre: Skipped 1 previous similar message [ 9712.654451] Lustre: server umount lustre-OST0000 complete [ 9719.271587] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9719.281808] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9719.297291] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9722.779694] Lustre: server umount lustre-MDT0000 complete [ 9726.434795] LustreError: 365623:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9726.453987] LustreError: 365623:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 9726.822257] LustreError: 365603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788823465 with bad export cookie 17057990664250364878 [ 9726.835880] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9727.061969] Lustre: server umount lustre-MDT0001 complete [ 9739.658259] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 9743.405975] Key type lgssc unregistered [ 9743.793956] LNet: 369158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9743.807760] LNetError: 369158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9743.823937] LNet: Removed LNI 192.168.201.128@tcp [ 9745.177226] Key type .llcrypt unregistered [ 9745.182593] Key type ._llcrypt unregistered [ 9771.602190] Key type ._llcrypt registered [ 9771.603307] Key type .llcrypt registered [ 9771.706762] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 9773.712364] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9773.898500] alg: No test for adler32 (adler32-zlib) [ 9775.171461] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 9775.484732] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 9777.248383] Key type lgssc registered [ 9779.616475] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9789.677576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9789.704240] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9789.715481] LustreError: 370596:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9789.756197] LustreError: 370596:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9789.767333] LustreError: 370596:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9789.783746] LustreError: 370596:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9789.797765] LustreError: 370596:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9797.891232] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9797.895272] LustreError: 370993:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9797.907436] LustreError: 370993:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9797.911940] LustreError: 370993:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9797.916795] LustreError: 370993:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9797.923420] LustreError: 370993:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9821.328780] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [ 9825.237455] Key type lgssc unregistered [ 9825.629948] LNet: 371997:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9825.640862] LNetError: 371997:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9825.667184] LNet: Removed LNI 192.168.201.128@tcp [ 9826.479399] Key type .llcrypt unregistered [ 9826.480881] Key type ._llcrypt unregistered [ 9844.929600] Key type ._llcrypt registered [ 9844.932696] Key type .llcrypt registered [ 9845.050934] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [ 9860.023772] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [ 9861.089139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9861.157852] alg: No test for adler32 (adler32-zlib) [ 9862.512253] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [ 9862.809658] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [ 9864.480216] Key type lgssc registered [ 9865.625499] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9903.723288] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9903.751097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9905.127676] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9905.148150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9905.243671] Lustre: lustre-MDT0000: new disk, initializing [ 9905.346509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9905.359351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9909.758287] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9920.043940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9920.238863] Lustre: 375167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9920.280869] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9920.284980] Lustre: Skipped 1 previous similar message [ 9920.348880] Lustre: lustre-MDT0001: new disk, initializing [ 9920.433829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9920.494132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9920.504841] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9925.430401] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9934.851917] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9940.674418] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9947.401084] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9947.635850] Lustre: lustre-OST0000: new disk, initializing [ 9947.639521] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9947.649336] Lustre: 377134:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9947.787234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9953.206661] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [ 9953.385897] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9953.400533] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9953.570783] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9962.294981] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9967.164558] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9967.509349] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9971.668777] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9971.837070] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9974.241460] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9974.247286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9974.255216] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9974.299250] Lustre: Skipped 1 previous similar message [ 9979.397395] Lustre: server umount lustre-OST0000 complete [ 9987.042808] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9987.058151] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9987.089024] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9987.109078] Lustre: Skipped 1 previous similar message [ 9989.601900] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9989.603794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9989.647142] Lustre: Skipped 1 previous similar message [ 9991.112783] Lustre: server umount lustre-MDT0000 complete [ 9996.251594] LustreError: 375160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788823735 with bad export cookie 8028723757426300778 [ 9996.254697] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9996.256293] LustreError: 375160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9996.459576] Lustre: server umount lustre-MDT0001 complete [10004.988302] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10007.004909] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:29:03 (1788823743) [10013.371101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10013.753535] LustreError: 379596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10013.773190] LustreError: 379596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10013.883815] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10018.380478] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10019.302190] LustreError: 379597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10023.400399] LustreError: 379596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10025.596875] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10025.964947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10029.644182] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10036.112851] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10043.057394] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10049.774029] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10050.055288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10055.745597] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10061.612114] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10068.484655] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10068.648905] Lustre: lustre-OST0001: new disk, initializing [10068.652641] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10068.660261] Lustre: 382429:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10068.730637] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10074.139543] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10074.766912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10074.780571] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10074.921483] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10082.638704] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10090.338439] Lustre: lustre-OST0001: Not available for connect from 192.168.201.28@tcp (stopping) [10090.476533] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10090.477974] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10090.498322] LustreError: Skipped 1 previous similar message [10090.519390] Lustre: Skipped 1 previous similar message [10092.973758] Lustre: server umount lustre-OST0001 complete [10095.416797] LustreError: 382817:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10100.718112] LustreError: 381419:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10100.735039] LustreError: 381419:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10110.945743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10110.955985] LustreError: 383355:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10110.961746] Lustre: Skipped 1 previous similar message [10110.962933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10110.962939] Lustre: Skipped 2 previous similar messages [10111.008983] LustreError: 383355:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10121.187831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10121.200627] Lustre: Skipped 4 previous similar messages [10124.256124] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10124.402160] Lustre: server umount lustre-OST0000 complete [10128.355077] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10128.363747] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10134.403460] Lustre: server umount lustre-MDT0000 complete [10136.549126] LustreError: 379593:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10136.579357] LustreError: 379593:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [10139.557692] LustreError: 379578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788823878 with bad export cookie 8028723757426301660 [10139.564268] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10139.568719] LustreError: 379578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10146.359870] Lustre: server umount lustre-MDT0001 complete [10158.877203] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [10162.047974] Key type lgssc unregistered [10162.390798] LNet: 384420:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10162.393700] LNetError: 384420:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10162.403900] LNet: Removed LNI 192.168.201.128@tcp [10163.383395] Key type .llcrypt unregistered [10163.384905] Key type ._llcrypt unregistered [10213.853590] Key type ._llcrypt registered [10213.857972] Key type .llcrypt registered [10213.952208] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [10215.177551] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10215.279962] alg: No test for adler32 (adler32-zlib) [10216.437478] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [10216.645532] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [10218.368209] Key type lgssc registered [10220.248154] Lustre: Echo OBD driver; http://www.lustre.org/ [10231.702725] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10231.741760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10233.276996] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10233.352040] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10233.540187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10238.985860] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10248.345724] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10248.446631] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10248.497553] Lustre: 387218:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10248.877151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10253.841821] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10261.804920] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10266.387825] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10269.153242] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10269.167411] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10269.198076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10271.207822] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10271.222442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10274.401031] Lustre: server umount lustre-MDT0000 complete [10277.568305] LustreError: 387209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788824016 with bad export cookie 4517620844159183532 [10277.578009] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10277.970818] Lustre: server umount lustre-MDT0001 complete [10287.206748] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 19:33:44 (1788824024) [10294.670625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10295.439947] LustreError: 389729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10295.455737] LustreError: 389729:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10295.501622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10300.562399] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10300.905151] LustreError: 389730:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10306.018554] LustreError: 389729:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10309.522302] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10309.987030] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10314.237995] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10321.694657] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10327.613361] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10333.388623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10333.556991] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10333.728430] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10340.115264] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10347.691991] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10354.436942] Lustre: Permanently deactivating lustre-OST0000 [10354.459503] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10362.883480] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10365.669738] LustreError: 392764:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10365.687732] LustreError: 392764:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10365.697831] LustreError: 392764:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10369.143056] LustreError: 392812:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10369.155483] LustreError: 392812:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10378.270619] Lustre: server umount lustre-OST0000 complete [10382.252333] LustreError: 393062:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10382.269623] LustreError: 393062:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10385.904736] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10385.916939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10385.923244] Lustre: Skipped 1 previous similar message [10385.949702] Lustre: Skipped 2 previous similar messages [10391.011147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10391.019890] Lustre: Skipped 1 previous similar message [10396.130773] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10396.143310] Lustre: Skipped 1 previous similar message [10399.203450] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10399.610740] Lustre: server umount lustre-MDT0000 complete [10401.250108] LustreError: 390763:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10401.262372] LustreError: 390763:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10403.570298] LustreError: 389709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788824142 with bad export cookie 4517620844159184106 [10403.577940] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10403.583706] LustreError: 389709:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [10414.251302] Lustre: server umount lustre-MDT0001 complete [10422.542854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10427.351135] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10430.909846] LustreError: 394223:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10438.085161] Lustre: 394565:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.128@tcp [10440.066367] Lustre: server umount MGS complete [10452.526128] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10452.876545] LustreError: 395290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10452.896827] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10452.899962] Lustre: Skipped 1 previous similar message [10452.940184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10457.782285] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10463.234839] LustreError: 395290:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10463.246181] LustreError: 395290:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10466.003231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10466.343326] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10470.824263] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10477.574518] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10482.914688] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10484.969994] Lustre: Permanently reactivating lustre-OST0000 [10484.983162] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10484.988180] Lustre: Skipped 2 previous similar messages [10515.426047] LustreError: 397169:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10524.467591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10524.875979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10524.881581] Lustre: Skipped 1 previous similar message [10532.877747] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10540.812575] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10545.643969] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10545.652867] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10545.666909] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10546.657964] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10547.955799] Lustre: server umount lustre-OST0000 complete [10553.313406] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10553.318182] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10553.326054] Lustre: Skipped 1 previous similar message [10558.588548] Lustre: server umount lustre-MDT0000 complete [10562.019237] LustreError: 395286:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10562.051698] LustreError: 395286:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10562.584691] LustreError: 395272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788824301 with bad export cookie 4517620844159185163 [10562.586825] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10562.592455] LustreError: 395272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10562.841450] Lustre: server umount lustre-MDT0001 complete [10573.431984] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [10576.558276] Key type lgssc unregistered [10576.788899] LNet: 399751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10576.805807] LNetError: 399751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10576.831341] LNet: Removed LNI 192.168.201.128@tcp [10577.585873] Key type .llcrypt unregistered [10577.606624] Key type ._llcrypt unregistered [10598.007689] Key type ._llcrypt registered [10598.012279] Key type .llcrypt registered [10598.150419] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [10613.818270] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [10614.892517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10615.023797] alg: No test for adler32 (adler32-zlib) [10616.173186] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [10616.388598] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [10618.042531] Key type lgssc registered [10619.215702] Lustre: Echo OBD driver; http://www.lustre.org/ [10654.653068] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10654.704209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10655.976415] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10656.050404] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10656.178685] Lustre: lustre-MDT0000: new disk, initializing [10656.293313] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10656.325517] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10661.242848] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10673.542380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10673.745353] Lustre: 402905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10673.850391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10673.865086] Lustre: Skipped 1 previous similar message [10673.984659] Lustre: lustre-MDT0001: new disk, initializing [10674.151472] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10674.198255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10674.210350] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10679.605354] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10688.660525] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10693.957690] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10700.943625] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10701.178712] Lustre: lustre-OST0000: new disk, initializing [10701.182448] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10701.191911] Lustre: 404875:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10701.304569] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10707.973798] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10708.108400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10708.123861] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10708.243811] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10717.928760] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10723.117920] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10723.325541] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10728.383864] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10728.635278] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10734.063396] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10734.073790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10734.087438] Lustre: Skipped 1 previous similar message [10734.097298] Lustre: Skipped 1 previous similar message [10737.045551] Lustre: server umount lustre-OST0000 complete [10740.705384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10740.715468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10740.739155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10744.292293] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10744.295223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10744.330977] Lustre: Skipped 1 previous similar message [10746.845587] Lustre: server umount lustre-MDT0000 complete [10751.351557] LustreError: 402895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788824490 with bad export cookie 17371893576343569004 [10751.370445] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10751.763421] Lustre: server umount lustre-MDT0001 complete [10759.821619] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10761.448585] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 19:41:38 (1788824498) [10768.200354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10768.572626] LustreError: 407335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10768.594365] LustreError: 407335:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10768.643303] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10773.019712] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10773.992690] LustreError: 407336:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10779.108871] LustreError: 407335:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10782.041832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10782.654645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10787.453938] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10794.839428] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10800.017562] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10805.546987] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10805.891542] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10812.819640] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10819.899338] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10821.784954] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10825.273286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10830.842153] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10830.859558] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10830.870571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10830.879615] Lustre: Skipped 1 previous similar message [10835.610652] Lustre: server umount lustre-OST0000 complete [10841.057382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10841.067941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10841.071898] Lustre: Skipped 2 previous similar messages [10841.087581] Lustre: Skipped 1 previous similar message [10846.120434] Lustre: server umount lustre-MDT0000 complete [10846.182897] LustreError: 407330:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10846.227105] LustreError: 407330:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10850.153404] LustreError: 407317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788824589 with bad export cookie 17371893576343569886 [10850.167697] LustreError: 407317:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10850.170158] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10850.556970] Lustre: server umount lustre-MDT0001 complete [10862.861232] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [10866.254671] Key type lgssc unregistered [10866.661350] LNet: 411061:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10866.672810] LNetError: 411061:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10866.694026] LNet: Removed LNI 192.168.201.128@tcp [10867.634846] Key type .llcrypt unregistered [10867.640728] Key type ._llcrypt unregistered [10894.486754] Key type ._llcrypt registered [10894.490302] Key type .llcrypt registered [10894.729687] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [10895.067616] Key type .llcrypt unregistered [10895.070185] Key type ._llcrypt unregistered [10914.333868] Key type ._llcrypt registered [10914.336897] Key type .llcrypt registered [10914.475197] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [10915.942909] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10916.081577] alg: No test for adler32 (adler32-zlib) [10917.243848] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [10917.560507] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [10919.192907] Key type lgssc registered [10920.475904] Lustre: Echo OBD driver; http://www.lustre.org/ [10928.436990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10928.481226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10930.062765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10935.642743] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10939.377828] LustreError: 413269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10939.392763] LustreError: 413269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10944.486325] LustreError: 413270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10944.742814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10945.331533] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10950.631206] LustreError: 413269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10950.658260] LustreError: 413269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10952.323483] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10954.723826] LustreError: 413270:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10954.750698] LustreError: 413270:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10954.767848] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10959.258483] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10959.841536] LustreError: 413269:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10959.858894] LustreError: 413269:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10965.891337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10966.315358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10970.532740] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [10979.807757] Lustre: server umount lustre-OST0000 complete [10985.955473] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10985.961081] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10985.984263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10986.979281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10986.981330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10992.097931] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10992.103916] Lustre: Skipped 1 previous similar message [10993.224487] Lustre: server umount lustre-MDT0000 complete [10996.530897] LustreError: 413251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788824735 with bad export cookie 1100870543957001929 [10996.538768] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10996.547386] LustreError: 413251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11007.147541] Lustre: server umount lustre-MDT0001 complete [11018.654335] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [11021.669913] Key type lgssc unregistered [11022.132039] LNet: 416416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11022.140687] LNetError: 416416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11022.166289] LNet: Removed LNI 192.168.201.128@tcp [11022.935387] Key type .llcrypt unregistered [11022.937636] Key type ._llcrypt unregistered [11050.336375] Key type ._llcrypt registered [11050.338757] Key type .llcrypt registered [11050.463473] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [11052.292144] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11052.317964] alg: No test for adler32 (adler32-zlib) [11053.399865] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [11053.675918] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [11055.360216] Key type lgssc registered [11056.764602] Lustre: Echo OBD driver; http://www.lustre.org/ [11065.120574] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11065.141781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11080.736597] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11085.856449] LustreError: 417804:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.128@tcp: failed processing log, type 4: rc = -110 [11112.545138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11119.006406] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11126.195863] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11132.484213] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11133.140356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11133.195939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11137.634042] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11138.539468] LustreError: 417830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11138.571343] LustreError: 417830:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11143.669722] LustreError: 417830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11145.558446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11145.918628] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11150.318215] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11158.278834] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11164.182623] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11171.809111] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11171.819045] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11171.832789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11173.859974] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11173.863905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11176.993572] Lustre: server umount lustre-OST0000 complete [11182.060056] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11182.072269] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11182.095238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11182.101051] Lustre: Skipped 1 previous similar message [11184.097966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11184.107182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11186.858836] Lustre: server umount lustre-MDT0000 complete [11189.219239] LustreError: 420538:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11189.244127] LustreError: 420538:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11190.193041] LustreError: 417812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788824929 with bad export cookie 7304332588811054862 [11190.197446] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11190.207589] LustreError: 417812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11190.506256] Lustre: server umount lustre-MDT0001 complete [11202.777299] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [11206.083722] Key type lgssc unregistered [11206.382510] LNet: 421375:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11206.390968] LNetError: 421375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11206.419291] LNet: Removed LNI 192.168.201.128@tcp [11207.559807] Key type .llcrypt unregistered [11207.562092] Key type ._llcrypt unregistered [11231.342737] Key type ._llcrypt registered [11231.345164] Key type .llcrypt registered [11231.470801] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [11232.659467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11232.802527] alg: No test for adler32 (adler32-zlib) [11233.972960] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [11234.322712] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [11235.968239] Key type lgssc registered [11237.179422] Lustre: Echo OBD driver; http://www.lustre.org/ [11245.630984] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11245.669742] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11247.219302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11251.664465] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11257.825510] LustreError: 422790:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11257.848789] LustreError: 422790:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11259.860170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11260.326504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11265.651494] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11273.408508] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11280.156497] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11287.107878] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11287.550609] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11292.668404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11294.474915] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11302.607155] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11309.404911] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.28@tcp (stopping) [11311.076496] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11311.084176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11311.092051] Lustre: Skipped 2 previous similar messages [11311.097462] Lustre: Skipped 1 previous similar message [11312.805876] Lustre: server umount lustre-MDT0000 complete [11314.499827] LustreError: 422787:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11315.018101] LustreError: 423524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.28@tcp arrived at 1788825053 with bad export cookie 1326056738882311180 [11315.035911] LustreError: 423524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11316.201630] LustreError: 423761:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11316.214806] LustreError: 423761:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11316.816241] LustreError: 422770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788825055 with bad export cookie 1326056738882310291 [11316.824726] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11316.829296] LustreError: 422770:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11321.313747] LustreError: 422790:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11321.325519] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11321.325835] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11321.326248] LustreError: 422790:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11321.327934] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11323.142967] Lustre: server umount lustre-OST0000 complete [11326.444069] LustreError: 422791:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11326.454494] LustreError: 422791:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11329.795790] Lustre: server umount lustre-MDT0001 complete [11340.098966] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [11342.976781] Key type lgssc unregistered [11343.282490] LNet: 426472:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11343.302134] LNetError: 426472:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11343.331021] LNet: Removed LNI 192.168.201.128@tcp [11344.462375] Key type .llcrypt unregistered [11344.465638] Key type ._llcrypt unregistered [11367.256273] Key type ._llcrypt registered [11367.258682] Key type .llcrypt registered [11367.354256] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [11368.288649] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11368.308661] alg: No test for adler32 (adler32-zlib) [11369.381521] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [11369.545210] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [11371.184148] Key type lgssc registered [11372.220623] Lustre: Echo OBD driver; http://www.lustre.org/ [11379.142689] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11379.156520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11380.520835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11384.917883] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11390.950395] LustreError: 427885:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11390.968032] LustreError: 427885:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11393.637181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11394.184439] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11398.914572] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11406.000988] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11411.771959] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11418.441416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11418.761698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11424.467300] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11425.979619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11431.332770] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11436.518660] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11436.520644] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11436.534867] Lustre: Skipped 1 previous similar message [11436.551163] Lustre: Skipped 1 previous similar message [11438.925338] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.28@tcp (stopping) [11441.636713] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11441.647123] Lustre: Skipped 1 previous similar message [11442.179368] Lustre: server umount lustre-MDT0001 complete [11444.024497] LustreError: 428917:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.28@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11446.757061] LustreError: 427882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11446.773094] LustreError: 427882:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11451.873521] LustreError: 427886:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11451.875572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11451.897275] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11454.207392] Lustre: server umount lustre-OST0000 complete [11457.008456] LustreError: 428476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11458.407701] Lustre: server umount lustre-MDT0000 complete [11472.409958] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [11474.779793] Key type lgssc unregistered [11475.108728] LNet: 431568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11475.119208] LNetError: 431568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11475.150914] LNet: Removed LNI 192.168.201.128@tcp [11476.093781] Key type .llcrypt unregistered [11476.096622] Key type ._llcrypt unregistered [11499.556751] Key type ._llcrypt registered [11499.559416] Key type .llcrypt registered [11499.643414] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [11513.330325] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [11514.432618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11514.471877] alg: No test for adler32 (adler32-zlib) [11515.537970] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [11515.741172] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [11517.376210] Key type lgssc registered [11518.569921] Lustre: Echo OBD driver; http://www.lustre.org/ [11553.111975] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [11567.626777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11567.661513] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11569.001721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11569.040930] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11569.152969] Lustre: lustre-MDT0000: new disk, initializing [11569.309555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11569.345847] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11573.526873] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11587.466418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11587.675419] Lustre: lustre-OST0000: new disk, initializing [11587.678625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11587.682086] Lustre: Skipped 1 previous similar message [11587.686199] Lustre: 436619:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11587.749257] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11593.687633] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11596.885944] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11596.900954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11596.929472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11602.623709] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11618.859968] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11618.971328] Lustre: 435601:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11619.054804] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11619.133526] Lustre: lustre-MDT0001: new disk, initializing [11619.286279] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11619.357465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11619.364660] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11624.100328] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11633.763725] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11633.976404] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11638.580344] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11638.827910] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11644.903029] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11644.906402] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11644.920838] Lustre: Skipped 1 previous similar message [11644.928551] Lustre: Skipped 1 previous similar message [11650.069565] Lustre: server umount lustre-MDT0001 complete [11655.141748] LustreError: 435614:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11655.165282] LustreError: 435614:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11660.273220] LustreError: 435609:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11660.294205] LustreError: 435609:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11665.377356] LustreError: 438224:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11665.407845] LustreError: 438224:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11668.450854] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11668.476599] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11668.500179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11668.505047] Lustre: Skipped 1 previous similar message [11670.498977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11670.499788] LustreError: 437948:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.624938] LustreError: 435615:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11675.626242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11680.740069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11682.272247] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11682.552315] Lustre: server umount lustre-OST0000 complete [11685.865780] LustreError: 435615:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11685.874784] LustreError: 435615:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11692.492563] Lustre: server umount lustre-MDT0000 complete [11706.425531] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [11709.274261] Key type lgssc unregistered [11709.634278] LNet: 440371:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11709.654800] LNetError: 440371:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11709.670931] LNet: Removed LNI 192.168.201.128@tcp [11710.611199] Key type .llcrypt unregistered [11710.621912] Key type ._llcrypt unregistered [11746.137567] Key type ._llcrypt registered [11746.139621] Key type .llcrypt registered [11746.240538] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [11746.499440] Key type .llcrypt unregistered [11746.500617] Key type ._llcrypt unregistered [11762.642200] Key type ._llcrypt registered [11762.643320] Key type .llcrypt registered [11762.716727] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [11763.771946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11763.998512] alg: No test for adler32 (adler32-zlib) [11764.970985] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [11765.160712] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [11766.816588] Key type lgssc registered [11767.826879] Lustre: Echo OBD driver; http://www.lustre.org/ [11791.630450] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11791.674301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11792.884675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11792.924525] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11792.979839] Lustre: lustre-MDT0000: new disk, initializing [11793.034642] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11793.049899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11796.781736] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11806.839398] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11807.036336] Lustre: 443738:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11807.177027] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11807.183338] Lustre: Skipped 1 previous similar message [11807.300337] Lustre: lustre-MDT0001: new disk, initializing [11807.402609] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11807.459233] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11807.475050] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11812.056793] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11820.660802] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11825.216711] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11830.971820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11831.198651] Lustre: lustre-OST0000: new disk, initializing [11831.203858] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11831.208929] Lustre: 445707:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11831.291400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11836.614363] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11837.554198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11837.570632] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11837.782313] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11845.232596] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11853.290882] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11853.306129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11853.309029] Lustre: Skipped 2 previous similar messages [11853.320765] Lustre: Skipped 2 previous similar messages [11856.453042] Lustre: server umount lustre-MDT0000 complete [11859.653613] LustreError: 443731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788825598 with bad export cookie 6705432622352053718 [11859.660640] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11859.664109] LustreError: 443731:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11860.058950] Lustre: server umount lustre-MDT0001 complete [11873.722719] Lustre: server umount lustre-OST0000 complete [11882.229769] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:00:19 (1788825619) [11888.826285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11889.201861] LustreError: 447735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11889.218366] LustreError: 447735:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11889.259583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11893.491493] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11894.754453] LustreError: 447736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11899.879147] LustreError: 447735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11902.236063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11902.580318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11907.446907] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11914.804732] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11920.433162] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11939.129929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11939.344371] Lustre: Found index 0 for lustre-OST0000, updating log [11939.499487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11945.354500] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [11951.812488] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11961.012772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11966.436844] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11966.443507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11966.455293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11967.526805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11967.531406] Lustre: Skipped 2 previous similar messages [11972.515443] Lustre: server umount lustre-OST0000 complete [11988.963939] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11988.971344] Lustre: Skipped 1 previous similar message [11988.976617] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11988.985446] Lustre: Skipped 1 previous similar message [11993.484385] Lustre: server umount lustre-MDT0000 complete [11994.084960] LustreError: 447736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11994.100031] LustreError: 447736:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11996.198471] LustreError: 447717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788825735 with bad export cookie 6705432622352055111 [11996.216500] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11996.216704] LustreError: 447717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11996.524826] Lustre: server umount lustre-MDT0001 complete [12002.692915] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:02:20 (1788825740) [12002.784302] LNet: 1 local NIs in recovery (showing 1): 192.168.201.128@tcp [12614.366377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12614.706456] LustreError: 454174:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12614.716181] LustreError: 454174:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12614.778334] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12619.317717] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [12620.259169] LustreError: 454175:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12624.374336] LustreError: 454174:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12627.468036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12628.011529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12632.055068] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [12638.990437] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12644.742796] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12665.030627] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12665.545816] Lustre: Found index 0 for lustre-OST0000, updating log [12665.866359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12669.479442] LNet: 1 local NIs in recovery (showing 1): 192.168.201.128@tcp [12672.839375] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [12681.001683] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12683.212970] Lustre: server umount lustre-OST0000 complete [12688.744328] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12688.752512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12688.757368] Lustre: Skipped 2 previous similar messages [12688.763590] Lustre: Skipped 1 previous similar message [12689.830176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12689.835566] Lustre: Skipped 1 previous similar message [12690.852560] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12690.864539] Lustre: Skipped 1 previous similar message [12692.901132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12692.911378] Lustre: Skipped 3 previous similar messages [12694.090657] Lustre: server umount lustre-MDT0000 complete [12694.957423] LustreError: 454169:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12694.986932] LustreError: 454169:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12697.805912] LustreError: 454156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788826436 with bad export cookie 6705432622352056189 [12697.819748] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12698.284508] Lustre: server umount lustre-MDT0001 complete [12728.964441] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [12729.888216] LNet: 1 local NIs in recovery (showing 1): 192.168.201.128@tcp [12739.757804] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [12778.769624] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:15:15 (1788826515) [12795.937368] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [12798.380409] Key type lgssc unregistered [12798.739564] LNet: 460791:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12798.748087] LNetError: 460791:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12928.808799] LNet: Removed LNI 192.168.201.128@tcp [12929.636756] Key type .llcrypt unregistered [12929.639733] Key type ._llcrypt unregistered [12944.185071] Key type ._llcrypt registered [12944.187426] Key type .llcrypt registered [12944.283581] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [12945.172953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12945.237818] alg: No test for adler32 (adler32-zlib) [12946.220631] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [12946.473558] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [12948.137176] Key type lgssc registered [12948.949141] Lustre: Echo OBD driver; http://www.lustre.org/ [12952.198651] LNet: Added LNI 192.168.201.128@tcp100 [8/256/0/180] [12952.900296] LNet: Added LNI 192.168.201.128@tcp101 [8/256/0/180] [12953.520151] LNet: Added LNI 192.168.201.128@tcp102 [8/256/0/180] [12954.162189] LNet: Added LNI 192.168.201.128@tcp103 [8/256/0/180] [12954.864640] LNet: Added LNI 192.168.201.128@tcp104 [8/256/0/180] [12955.566143] LNet: Added LNI 192.168.201.128@tcp105 [8/256/0/180] [12956.162535] LNet: Added LNI 192.168.201.128@tcp106 [8/256/0/180] [12956.837985] LNet: Added LNI 192.168.201.128@tcp107 [8/256/0/180] [12957.500582] LNet: Added LNI 192.168.201.128@tcp108 [8/256/0/180] [12958.268778] LNet: Added LNI 192.168.201.128@tcp109 [8/256/0/180] [12958.964595] LNet: Added LNI 192.168.201.128@tcp110 [8/256/0/180] [12959.646697] LNet: Added LNI 192.168.201.128@tcp111 [8/256/0/180] [12960.267782] LNet: Added LNI 192.168.201.128@tcp112 [8/256/0/180] [12961.001688] LNet: Added LNI 192.168.201.128@tcp113 [8/256/0/180] [12961.649033] LNet: Added LNI 192.168.201.128@tcp114 [8/256/0/180] [12962.287036] LNet: Added LNI 192.168.201.128@tcp115 [8/256/0/180] [12962.979037] LNet: Added LNI 192.168.201.128@tcp116 [8/256/0/180] [12963.684487] LNet: Added LNI 192.168.201.128@tcp117 [8/256/0/180] [12964.301844] LNet: Added LNI 192.168.201.128@tcp118 [8/256/0/180] [12964.929188] LNet: Added LNI 192.168.201.128@tcp119 [8/256/0/180] [12965.561929] LNet: Added LNI 192.168.201.128@tcp120 [8/256/0/180] [12966.267714] LNet: Added LNI 192.168.201.128@tcp121 [8/256/0/180] [12967.109859] LNet: Added LNI 192.168.201.128@tcp122 [8/256/0/180] [12967.855355] LNet: Added LNI 192.168.201.128@tcp123 [8/256/0/180] [12968.540986] LNet: Added LNI 192.168.201.128@tcp124 [8/256/0/180] [12969.330331] LNet: Added LNI 192.168.201.128@tcp125 [8/256/0/180] [12970.163621] LNet: Added LNI 192.168.201.128@tcp126 [8/256/0/180] [12970.943930] LNet: Added LNI 192.168.201.128@tcp127 [8/256/0/180] [12971.811731] LNet: Added LNI 192.168.201.128@tcp128 [8/256/0/180] [12972.465208] LNet: Added LNI 192.168.201.128@tcp129 [8/256/0/180] [12973.058270] LNet: Added LNI 192.168.201.128@tcp130 [8/256/0/180] [12973.683393] LNet: Added LNI 192.168.201.128@tcp131 [8/256/0/180] [12974.358700] LNet: Added LNI 192.168.201.128@tcp132 [8/256/0/180] [12975.036637] LNet: Added LNI 192.168.201.128@tcp133 [8/256/0/180] [12975.654962] LNet: Added LNI 192.168.201.128@tcp134 [8/256/0/180] [12976.364606] LNet: Added LNI 192.168.201.128@tcp135 [8/256/0/180] [12981.781138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12981.805057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12982.932646] Lustre: 463821:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.128@tcp: can't add NID 192.168.201.128@tcp130: rc = -75 [12983.033619] LNetError: 463843:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.128 [12983.050496] LNetError: Protocol error connecting to 192.168.201.128@tcp115 at host 192.168.201.128:988: is it running a compatible version of Lustre? [12983.059482] LNetError: Skipped 3 previous similar messages [12983.216363] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12983.241340] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12983.308526] Lustre: lustre-MDT0000: new disk, initializing [12983.333355] LNet: 4 local NIs in recovery (showing 4): 192.168.201.128@tcp123, 192.168.201.128@tcp124, 192.168.201.128@tcp126, 192.168.201.128@tcp128 [12983.393762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12983.406858] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12984.099273] LNetError: 463841:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.128 [12984.107561] LNetError: 463841:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [12984.115967] LNetError: Protocol error connecting to 192.168.201.128@tcp116 at host 192.168.201.128:988: is it running a compatible version of Lustre? [12984.126468] LNetError: Skipped 5 previous similar messages [12986.401078] LNetError: 463853:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.128 [12986.413649] LNetError: 463853:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [12986.428319] LNetError: Protocol error connecting to 192.168.201.128@tcp123 at host 192.168.201.128:988: is it running a compatible version of Lustre? [12986.439110] LNetError: Skipped 4 previous similar messages [12987.521492] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [12990.498958] LNetError: 463842:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.128 [12990.504959] LNetError: Protocol error connecting to 192.168.201.128@tcp123 at host 192.168.201.128:988: is it running a compatible version of Lustre? [12995.938368] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12996.085180] Lustre: 463839:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12996.141775] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12996.145343] Lustre: Skipped 1 previous similar message [12996.207550] Lustre: lustre-MDT0001: new disk, initializing [12996.241750] Lustre: 464641:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.128@tcp: can't add NID 192.168.201.128@tcp130: rc = -75 [12996.246751] Lustre: 464641:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12996.344139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12996.389471] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12996.408866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12998.692911] LNetError: 461278:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.128 [12998.700240] LNetError: Protocol error connecting to 192.168.201.128@tcp123 at host 192.168.201.128:988: is it running a compatible version of Lustre? [12999.924212] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13001.240640] Lustre: 465003:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.128@tcp: can't add NID 192.168.201.128@tcp130: rc = -75 [13001.249870] Lustre: 465003:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13006.940584] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13010.913700] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13012.008517] LNetError: 463843:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.128 [13012.019085] LNetError: Protocol error connecting to 192.168.201.128@tcp103 at host 192.168.201.128:988: is it running a compatible version of Lustre? [13015.957349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13016.192723] Lustre: lustre-OST0000: new disk, initializing [13016.196354] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13016.201842] Lustre: 465811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13016.268192] Lustre: 465816:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.128@tcp: can't add NID 192.168.201.128@tcp130: rc = -75 [13016.272296] Lustre: 465816:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13016.303713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13020.815755] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13025.833767] Lustre: 466414:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.128@tcp: can't add NID 192.168.201.128@tcp130: rc = -75 [13025.846736] Lustre: 466414:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13025.910996] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13025.916740] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13026.039106] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13028.338427] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13029.411518] LNetError: 463841:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.128 [13029.427236] LNetError: 463841:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [13029.436216] LNetError: Protocol error connecting to 192.168.201.128@tcp125 at host 192.168.201.128:988: is it running a compatible version of Lustre? [13029.441250] LNetError: Skipped 10 previous similar messages [13036.513212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13036.514334] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13036.521215] LustreError: Skipped 1 previous similar message [13036.526125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13036.542226] Lustre: Skipped 1 previous similar message [13036.558887] Lustre: Skipped 1 previous similar message [13039.601246] Lustre: server umount lustre-OST0000 complete [13042.658267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13042.664059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13042.676647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13043.746336] LNet: 1 local NIs in recovery (showing 1): 192.168.201.128@tcp123 [13046.758308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13046.760538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13046.775628] Lustre: Skipped 1 previous similar message [13048.441186] Lustre: server umount lustre-MDT0000 complete [13051.357743] LustreError: 463830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788826790 with bad export cookie 12786895166086213810 [13051.370528] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13051.716950] Lustre: server umount lustre-MDT0001 complete [13059.444232] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [13061.473186] Key type lgssc unregistered [13061.807168] LNet: 467724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13061.816291] LNetError: 467724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13061.819264] LNet: Removed LNI 192.168.201.128@tcp [13062.880181] LNet: Removed LNI 192.168.201.128@tcp100 [13062.883712] LNetError: 467724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13062.890256] LNetError: 467724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13062.897322] LNet: Removed LNI 192.168.201.128@tcp101 [13063.904821] LNet: Removed LNI 192.168.201.128@tcp102 [13063.914895] LNetError: 467724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13063.920703] LNetError: 467724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13063.929471] LNet: Removed LNI 192.168.201.128@tcp103 [13064.992179] LNet: Removed LNI 192.168.201.128@tcp104 [13066.016145] LNet: Removed LNI 192.168.201.128@tcp105 [13066.019551] LNetError: 467724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13066.029051] LNetError: 467724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13066.038985] LNet: Removed LNI 192.168.201.128@tcp106 [13067.104341] LNet: Removed LNI 192.168.201.128@tcp107 [13068.129832] LNet: Removed LNI 192.168.201.128@tcp108 [13069.152273] LNet: Removed LNI 192.168.201.128@tcp109 [13070.177887] LNet: Removed LNI 192.168.201.128@tcp110 [13070.183496] LNetError: 467724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13070.191408] LNetError: 467724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13070.196839] LNet: Removed LNI 192.168.201.128@tcp111 [13071.264481] LNet: Removed LNI 192.168.201.128@tcp112 [13072.288655] LNet: Removed LNI 192.168.201.128@tcp113 [13073.312971] LNet: Removed LNI 192.168.201.128@tcp114 [13074.336221] LNet: Removed LNI 192.168.201.128@tcp115 [13075.360194] LNet: Removed LNI 192.168.201.128@tcp116 [13076.448474] LNet: Removed LNI 192.168.201.128@tcp117 [13077.473424] LNet: Removed LNI 192.168.201.128@tcp118 [13078.496196] LNet: Removed LNI 192.168.201.128@tcp119 [13078.498194] LNetError: 467724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13078.508901] LNetError: 467724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13078.516666] LNet: Removed LNI 192.168.201.128@tcp120 [13079.584783] LNet: Removed LNI 192.168.201.128@tcp121 [13080.609848] LNet: Removed LNI 192.168.201.128@tcp122 [13080.612383] LNet: Removed LNI 192.168.201.128@tcp123 [13081.634094] LNet: Removed LNI 192.168.201.128@tcp124 [13082.656180] LNet: Removed LNI 192.168.201.128@tcp125 [13083.680594] LNet: Removed LNI 192.168.201.128@tcp126 [13084.704184] LNet: Removed LNI 192.168.201.128@tcp127 [13085.728253] LNet: Removed LNI 192.168.201.128@tcp128 [13086.752198] LNet: Removed LNI 192.168.201.128@tcp129 [13086.756347] LNet: Removed LNI 192.168.201.128@tcp130 [13086.759466] LNet: Removed LNI 192.168.201.128@tcp131 [13086.762990] LNet: Removed LNI 192.168.201.128@tcp132 [13086.768401] LNet: Removed LNI 192.168.201.128@tcp133 [13086.771431] LNet: Removed LNI 192.168.201.128@tcp134 [13086.783472] LNet: Removed LNI 192.168.201.128@tcp135 [13087.444895] Key type .llcrypt unregistered [13087.447266] Key type ._llcrypt unregistered [13104.410469] Key type ._llcrypt registered [13104.413419] Key type .llcrypt registered [13104.496864] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [13114.897499] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [13115.967882] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13116.006327] alg: No test for adler32 (adler32-zlib) [13117.011705] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [13117.245977] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [13118.880279] Key type lgssc registered [13119.768534] Lustre: Echo OBD driver; http://www.lustre.org/ [13145.958545] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:21:23 (1788826883) [13161.059390] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [13163.369948] Key type lgssc unregistered [13163.670632] LNet: 471785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13163.675699] LNetError: 471785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13163.690908] LNet: Removed LNI 192.168.201.128@tcp [13164.508709] Key type .llcrypt unregistered [13164.510093] Key type ._llcrypt unregistered [13177.218096] Key type ._llcrypt registered [13177.219609] Key type .llcrypt registered [13177.310824] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [13178.103167] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13178.214601] alg: No test for adler32 (adler32-zlib) [13179.191848] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [13179.411630] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [13181.072264] Key type lgssc registered [13181.764109] Lustre: Echo OBD driver; http://www.lustre.org/ [13188.417045] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13188.428391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13189.644828] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13189.665322] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13189.713531] Lustre: lustre-MDT0000: new disk, initializing [13189.789605] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13189.799964] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13193.822769] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13201.946515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13202.054837] Lustre: 472972:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13202.146853] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13202.150803] Lustre: Skipped 1 previous similar message [13202.248161] Lustre: lustre-MDT0001: new disk, initializing [13202.306366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13202.325842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13202.331863] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13205.378661] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13212.473414] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13216.124765] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13220.866617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13221.047473] Lustre: lustre-OST0000: new disk, initializing [13221.050206] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13221.058277] Lustre: 474939:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13221.123626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13225.747522] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13230.143572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13230.157918] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13230.239606] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13232.965284] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13236.049749] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing lnet_if_list [13239.866552] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13239.892791] LNet: Added LNI 192.168.201.128@tcp10 [8/256/0/180] [13243.847746] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13243.868326] LNet: Added LNI 192.168.201.128@tcp11 [8/256/0/180] [13247.187558] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13247.209348] LNet: Added LNI 192.168.201.128@tcp12 [8/256/0/180] [13251.001709] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13251.022755] LNet: Added LNI 192.168.201.128@tcp13 [8/256/0/180] [13254.644707] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13254.674165] LNet: Added LNI 192.168.201.128@tcp14 [8/256/0/180] [13258.465587] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13258.488444] LNet: Added LNI 192.168.201.128@tcp15 [8/256/0/180] [13262.340959] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13262.356896] LNet: Added LNI 192.168.201.128@tcp16 [8/256/0/180] [13266.159207] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13266.192901] LNet: Added LNI 192.168.201.128@tcp17 [8/256/0/180] [13270.409435] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13270.432897] LNet: Added LNI 192.168.201.128@tcp18 [8/256/0/180] [13274.077858] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13274.104590] LNet: Added LNI 192.168.201.128@tcp19 [8/256/0/180] [13277.750751] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13277.779821] LNet: Added LNI 192.168.201.128@tcp20 [8/256/0/180] [13282.321757] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13282.355992] LNet: Added LNI 192.168.201.128@tcp21 [8/256/0/180] [13286.572870] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13286.600743] LNet: Added LNI 192.168.201.128@tcp22 [8/256/0/180] [13290.805324] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13290.826553] LNet: Added LNI 192.168.201.128@tcp23 [8/256/0/180] [13295.914121] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13295.932251] LNet: Added LNI 192.168.201.128@tcp24 [8/256/0/180] [13300.677352] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13300.696312] LNet: Added LNI 192.168.201.128@tcp25 [8/256/0/180] [13305.484527] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13305.510675] LNet: Added LNI 192.168.201.128@tcp26 [8/256/0/180] [13310.003985] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13310.020760] LNet: Added LNI 192.168.201.128@tcp27 [8/256/0/180] [13314.917429] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13314.935837] LNet: Added LNI 192.168.201.128@tcp28 [8/256/0/180] [13319.958430] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13319.992894] LNet: Added LNI 192.168.201.128@tcp29 [8/256/0/180] [13324.746469] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13324.783381] LNet: Added LNI 192.168.201.128@tcp30 [8/256/0/180] [13330.247762] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13330.276223] LNet: Added LNI 192.168.201.128@tcp31 [8/256/0/180] [13335.412616] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13335.436074] LNet: Added LNI 192.168.201.128@tcp32 [8/256/0/180] [13340.398822] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13340.419061] LNet: Added LNI 192.168.201.128@tcp33 [8/256/0/180] [13345.141994] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13345.171629] LNet: Added LNI 192.168.201.128@tcp34 [8/256/0/180] [13350.442770] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13350.460536] LNet: Added LNI 192.168.201.128@tcp35 [8/256/0/180] [13355.715672] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13355.740409] LNet: Added LNI 192.168.201.128@tcp36 [8/256/0/180] [13360.345340] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13360.363906] LNet: Added LNI 192.168.201.128@tcp37 [8/256/0/180] [13364.204178] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13364.249618] LNet: Added LNI 192.168.201.128@tcp38 [8/256/0/180] [13369.411086] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13369.429742] LNet: Added LNI 192.168.201.128@tcp39 [8/256/0/180] [13374.215911] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13374.239972] LNet: Added LNI 192.168.201.128@tcp40 [8/256/0/180] [13379.146579] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13379.176764] LNet: Added LNI 192.168.201.128@tcp41 [8/256/0/180] [13383.766476] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13383.788253] LNet: Added LNI 192.168.201.128@tcp42 [8/256/0/180] [13388.221369] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13388.245420] LNet: Added LNI 192.168.201.128@tcp43 [8/256/0/180] [13393.040263] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13393.064559] LNet: Added LNI 192.168.201.128@tcp44 [8/256/0/180] [13398.109845] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13398.126986] LNet: Added LNI 192.168.201.128@tcp45 [8/256/0/180] [13402.875531] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13402.889614] LNet: Added LNI 192.168.201.128@tcp46 [8/256/0/180] [13408.214543] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13408.229756] LNet: Added LNI 192.168.201.128@tcp47 [8/256/0/180] [13413.951944] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13413.968898] LNet: Added LNI 192.168.201.128@tcp48 [8/256/0/180] [13419.739728] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13419.762422] LNet: Added LNI 192.168.201.128@tcp49 [8/256/0/180] [13426.120819] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13426.144504] LNet: Added LNI 192.168.201.128@tcp50 [8/256/0/180] [13440.482883] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13440.485829] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13440.505479] Lustre: Skipped 2 previous similar messages [13440.519521] Lustre: Skipped 2 previous similar messages [13445.605132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13445.612768] Lustre: Skipped 2 previous similar messages [13446.518318] Lustre: server umount lustre-MDT0000 complete [13449.005185] LustreError: 472964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788827187 with bad export cookie 16271357546533052572 [13449.008648] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13449.015109] LustreError: 472964:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13449.227927] Lustre: server umount lustre-MDT0001 complete [13461.112677] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [13466.050941] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13467.104551] Lustre: 472244:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788827189/real 1788827189] req@0000000029dc0b4d x1875720974501248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788827205 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13467.125154] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13470.176146] Lustre: 472243:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788827193/real 1788827193] req@0000000020b5e15c x1875720974501376/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788827209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13471.200153] Lustre: 472246:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788827194/real 1788827194] req@0000000021649269 x1875720974501504/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788827210 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13473.519787] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13474.538150] Lustre: 472246:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788827197/real 1788827197] req@000000006bf7a0f7 x1875720974501760/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788827213 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13476.385243] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13479.905466] Lustre: 472243:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788827202/real 1788827202] req@000000003a4f495e x1875720974501888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788827218 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13482.178592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13482.411992] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13485.466821] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13487.592287] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13490.063589] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13493.718820] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13495.783591] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13506.023356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13506.047050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13506.050591] Lustre: Skipped 1 previous similar message [13507.657213] Lustre: server umount lustre-OST0000 complete [13511.142856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13511.156491] Lustre: Skipped 1 previous similar message [13511.161463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13516.256962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13516.266986] Lustre: Skipped 2 previous similar messages [13516.633586] Lustre: server umount lustre-MDT0000 complete [13519.150897] LustreError: 472963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788827258 with bad export cookie 16271357546533054987 [13519.157466] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13519.160814] LustreError: 472963:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13519.370572] Lustre: server umount lustre-MDT0001 complete [13527.031106] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [13528.991347] Key type lgssc unregistered [13529.294188] LNet: 501916:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13529.302910] LNetError: 501916:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13529.307203] LNet: Removed LNI 192.168.201.128@tcp [13529.309569] LNet: Removed LNI 192.168.201.128@tcp10 [13529.310881] LNet: Removed LNI 192.168.201.128@tcp11 [13529.317209] LNet: Removed LNI 192.168.201.128@tcp12 [13529.320282] LNet: Removed LNI 192.168.201.128@tcp13 [13529.322268] LNet: Removed LNI 192.168.201.128@tcp14 [13529.325316] LNet: Removed LNI 192.168.201.128@tcp15 [13529.328551] LNet: Removed LNI 192.168.201.128@tcp16 [13529.330898] LNet: Removed LNI 192.168.201.128@tcp17 [13529.334883] LNet: Removed LNI 192.168.201.128@tcp18 [13529.337461] LNet: Removed LNI 192.168.201.128@tcp19 [13529.340416] LNet: Removed LNI 192.168.201.128@tcp20 [13529.343383] LNet: Removed LNI 192.168.201.128@tcp21 [13529.345724] LNet: Removed LNI 192.168.201.128@tcp22 [13529.348145] LNet: Removed LNI 192.168.201.128@tcp23 [13529.350292] LNet: Removed LNI 192.168.201.128@tcp24 [13529.352366] LNet: Removed LNI 192.168.201.128@tcp25 [13529.354789] LNet: Removed LNI 192.168.201.128@tcp26 [13529.357407] LNet: Removed LNI 192.168.201.128@tcp27 [13529.359759] LNet: Removed LNI 192.168.201.128@tcp28 [13529.362313] LNet: Removed LNI 192.168.201.128@tcp29 [13529.364668] LNet: Removed LNI 192.168.201.128@tcp30 [13529.366469] LNet: Removed LNI 192.168.201.128@tcp31 [13529.368421] LNet: Removed LNI 192.168.201.128@tcp32 [13529.370026] LNet: Removed LNI 192.168.201.128@tcp33 [13529.371777] LNet: Removed LNI 192.168.201.128@tcp34 [13529.373767] LNet: Removed LNI 192.168.201.128@tcp35 [13529.375597] LNet: Removed LNI 192.168.201.128@tcp36 [13529.377157] LNet: Removed LNI 192.168.201.128@tcp37 [13529.378926] LNet: Removed LNI 192.168.201.128@tcp38 [13529.381055] LNet: Removed LNI 192.168.201.128@tcp39 [13529.382670] LNet: Removed LNI 192.168.201.128@tcp40 [13529.383802] LNet: Removed LNI 192.168.201.128@tcp41 [13529.384822] LNet: Removed LNI 192.168.201.128@tcp42 [13529.386664] LNet: Removed LNI 192.168.201.128@tcp43 [13529.387895] LNet: Removed LNI 192.168.201.128@tcp44 [13529.389208] LNet: Removed LNI 192.168.201.128@tcp45 [13529.391067] LNet: Removed LNI 192.168.201.128@tcp46 [13529.392643] LNet: Removed LNI 192.168.201.128@tcp47 [13529.395137] LNet: Removed LNI 192.168.201.128@tcp48 [13529.396917] LNet: Removed LNI 192.168.201.128@tcp49 [13529.405262] LNet: Removed LNI 192.168.201.128@tcp50 [13530.034353] Key type .llcrypt unregistered [13530.036183] Key type ._llcrypt unregistered [13542.994331] Key type ._llcrypt registered [13542.997355] Key type .llcrypt registered [13543.081800] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [13552.678120] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [13553.428504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13553.488301] alg: No test for adler32 (adler32-zlib) [13554.448804] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [13554.644362] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [13556.280147] Key type lgssc registered [13556.991406] Lustre: Echo OBD driver; http://www.lustre.org/ [13580.377387] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13580.410541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13581.543835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13581.566444] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13581.625981] Lustre: lustre-MDT0000: new disk, initializing [13581.686212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13581.700242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13584.609532] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13591.677753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13591.752115] Lustre: 505095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13591.819781] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13591.825743] Lustre: Skipped 1 previous similar message [13591.892311] Lustre: lustre-MDT0001: new disk, initializing [13591.961637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13591.990325] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13591.998875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13594.699593] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13600.510463] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13603.955324] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13607.993208] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13608.076479] Lustre: *** cfs_fail_loc=90f, val=0*** [13608.120156] Lustre: lustre-OST0000: new disk, initializing [13608.122939] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13608.127412] Lustre: 507114:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13608.164375] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13611.427960] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13615.154088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13615.169898] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13615.235934] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13617.959478] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13625.316843] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13625.326365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13625.332511] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13625.830036] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13625.830228] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13625.847117] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13628.548601] Lustre: server umount lustre-OST0000 complete [13630.945657] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13630.956055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13630.963640] Lustre: Skipped 1 previous similar message [13636.065730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13636.072732] Lustre: Skipped 1 previous similar message [13641.187355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13641.197581] Lustre: Skipped 1 previous similar message [13645.280564] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13645.475560] Lustre: server umount lustre-MDT0000 complete [13647.911791] LustreError: 505087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788827386 with bad export cookie 12386128616547713618 [13647.920232] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13647.922113] LustreError: 505087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13648.146779] Lustre: server umount lustre-MDT0001 complete [13655.037334] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [13657.046909] Key type lgssc unregistered [13657.231786] LNet: 509016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13657.238883] LNetError: 509016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13658.281126] LNet: Removed LNI 192.168.201.128@tcp [13658.845770] Key type .llcrypt unregistered [13658.851060] Key type ._llcrypt unregistered [13671.584493] Key type ._llcrypt registered [13671.586767] Key type .llcrypt registered [13671.667827] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [13672.436992] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13672.458105] alg: No test for adler32 (adler32-zlib) [13673.483133] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [13673.668958] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [13675.307339] Key type lgssc registered [13676.117102] Lustre: Echo OBD driver; http://www.lustre.org/ [13681.503426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13681.517909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13682.852379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13685.797143] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13691.286895] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13691.543921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13694.322516] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13699.200818] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13703.307291] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13708.052297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13708.298084] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13712.485079] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13717.570718] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13735.225748] loop0: detected capacity change from 0 to 204800000 [13735.388596] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13735.395219] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13735.402280] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13738.929081] loop0: detected capacity change from 0 to 204800000 [13739.028916] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13739.034354] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13739.042278] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13742.151102] loop0: detected capacity change from 0 to 204800000 [13745.468345] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13745.584264] Lustre: fs15246-OST0000: new disk, initializing [13745.586993] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13745.594907] Lustre: 513873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13745.634894] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13749.832524] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13756.240858] loop1: detected capacity change from 0 to 204800000 [13760.002636] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13760.104602] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13760.139877] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13760.204784] Lustre: fs15246-MDT0000: new disk, initializing [13760.329838] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13760.352688] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13760.419150] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13761.443608] Lustre: 514994:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000893b757f x1875721492406272/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788827516 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13763.398415] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13766.498831] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13766.503829] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13766.516396] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13781.478092] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13781.498697] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13783.621204] Lustre: server umount fs15246-MDT0000 complete [13786.601398] LustreError: 510257:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13786.616141] LustreError: 510257:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13789.192618] Lustre: server umount fs15246-OST0000 complete [13796.838478] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13796.839461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13796.847270] Lustre: Skipped 1 previous similar message [13796.856026] Lustre: Skipped 1 previous similar message [13801.108633] Lustre: server umount lustre-OST0000 complete [13804.518458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13804.521904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13804.528116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13807.073994] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13807.074467] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13809.627787] Lustre: server umount lustre-MDT0000 complete [13811.781483] LustreError: 510240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788827550 with bad export cookie 2501969300789751858 [13811.784806] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13811.787605] LustreError: 510240:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13812.024251] Lustre: server umount lustre-MDT0001 complete [13819.236243] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [13821.014117] Key type lgssc unregistered [13821.261090] LNet: 518087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13821.268093] LNetError: 518087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13821.281817] LNet: Removed LNI 192.168.201.128@tcp [13821.906550] Key type .llcrypt unregistered [13821.908245] Key type ._llcrypt unregistered [13833.363214] Key type ._llcrypt registered [13833.364737] Key type .llcrypt registered [13833.422686] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [13834.040438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13834.049654] alg: No test for adler32 (adler32-zlib) [13835.018047] Lustre: Lustre: Build Version: 2.17.58_40_gea5fd3f [13835.189244] LNet: Added LNI 192.168.201.128@tcp [8/256/0/180] [13836.800245] Key type lgssc registered [13837.462286] Lustre: Echo OBD driver; http://www.lustre.org/ [13861.449802] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [13867.078538] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [13886.212800] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13886.221887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13887.396372] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13887.414202] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13887.453281] Lustre: lustre-MDT0000: new disk, initializing [13887.503587] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13887.518159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13889.814133] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13895.142413] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13895.230564] Lustre: 522520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13895.274769] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13895.277239] Lustre: Skipped 1 previous similar message [13895.323733] Lustre: lustre-MDT0001: new disk, initializing [13895.381361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13895.409144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13895.414120] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13897.972269] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13903.561313] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13906.435767] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13910.542153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13910.763755] Lustre: lustre-OST0000: new disk, initializing [13910.766938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13910.772231] Lustre: 524487:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13910.834062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13911.507948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13911.519314] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13911.640892] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13914.651759] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_default_debug -1 all [13920.812456] Lustre: DEBUG MARKER: oleg128-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13923.330136] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13923.448061] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13925.473356] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13925.606897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13926.883448] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13926.890097] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13926.895120] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13926.906680] Lustre: Skipped 1 previous similar message [13932.001636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13932.004884] Lustre: Skipped 1 previous similar message [13937.130421] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13937.140196] Lustre: Skipped 1 previous similar message [13941.216239] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13941.528929] Lustre: server umount lustre-OST0000 complete [13951.969761] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13951.985316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13952.012346] Lustre: Skipped 1 previous similar message [13952.027256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13955.812854] Lustre: server umount lustre-MDT0000 complete [13960.261314] LustreError: 522514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788827699 with bad export cookie 14156530717395347842 [13960.264552] LustreError: MGC192.168.201.128@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13960.269754] LustreError: 522514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13960.651432] Lustre: server umount lustre-MDT0001 complete [14002.105623] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing set_hostid [14016.785926] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing load_modules_local [14045.486734] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13781 sec ========= 20:36:22 (1788827782) [14046.863986] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 20:36:24 (1788827784) === [14049.987452] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 20:36:27 (1788827787) === [14086.728784] Lustre: DEBUG MARKER: oleg128-server.virtnet: executing unload_modules_local [14089.544188] Key type lgssc unregistered [14089.889477] LNet: 531153:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14089.897765] LNetError: 531153:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14089.931178] LNet: Removed LNI 192.168.201.128@tcp [14091.037783] Key type .llcrypt unregistered [14091.040370] Key type ._llcrypt unregistered