[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 490480431 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001010] APIC: Switch to symmetric I/O mode setup [ 0.002337] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004013] kvm-guest: setup PV IPIs [ 0.006906] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.010076] LSM: Security Framework initializing [ 0.011067] Yama: becoming mindful. [ 0.012040] SELinux: Initializing. [ 0.013064] *** VALIDATE selinux *** [ 0.021445] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025573] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026112] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027105] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028110] *** VALIDATE tmpfs *** [ 0.029437] *** VALIDATE proc *** [ 0.031153] *** VALIDATE cgroup *** [ 0.032000] *** VALIDATE cgroup2 *** [ 0.032282] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034146] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.037005] Spectre V2 : User space: Vulnerable [ 0.038008] Speculative Store Bypass: Vulnerable [ 0.041320] debug: unmapping init [mem 0xffffffff8ba59000-0xffffffff8ba60fff] [ 0.043835] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044741] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045022] ... version: 2 [ 0.046012] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049014] ... max period: 00007fffffffffff [ 0.050020] ... fixed-purpose events: 3 [ 0.051022] ... event mask: 000000070000000f [ 0.052291] rcu: Hierarchical SRCU implementation. [ 0.054479] smp: Bringing up secondary CPUs ... [ 0.055556] x86: Booting SMP configuration: [ 0.056024] .... node #0, CPUs: #1 #2 #3 [ 0.059170] smp: Brought up 1 node, 4 CPUs [ 0.061012] smpboot: Max logical packages: 1 [ 0.062029] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.133304] node 0 deferred pages initialised in 68ms [ 0.136447] devtmpfs: initialized [ 0.138263] x86/mm: Memory block size: 128MB [ 0.141895] gcov: version magic: 0x41383552 [ 0.144354] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.145072] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.147292] pinctrl core: initialized pinctrl subsystem [ 0.149251] [ 0.149793] ************************************************************* [ 0.153012] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155010] ** ** [ 0.157011] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.160014] ** ** [ 0.163012] ** This means that this kernel is built to expose internal ** [ 0.165010] ** IOMMU data structures, which may compromise security on ** [ 0.167011] ** your system. ** [ 0.170028] ** ** [ 0.173013] ** If you see this message and you are not debugging the ** [ 0.176011] ** kernel, report this immediately to your vendor! ** [ 0.178009] ** ** [ 0.180011] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.183013] ************************************************************* [ 0.186657] NET: Registered protocol family 16 [ 0.188439] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.191077] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195067] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199012] cpuidle: using governor menu [ 0.200766] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.204115] PCI: Using configuration type 1 for base access [ 0.206273] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.215140] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.216044] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.218082] cryptd: max_cpu_qlen set to 1000 [ 0.220270] ACPI: Added _OSI(Module Device) [ 0.221012] ACPI: Added _OSI(Processor Device) [ 0.222010] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.224014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.230594] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.236360] ACPI: Interpreter enabled [ 0.238059] ACPI: PM: (supports S0 S3 S4 S5) [ 0.239010] ACPI: Using IOAPIC for interrupt routing [ 0.241120] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.245402] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.254744] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.257034] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.259017] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.263077] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.268285] acpiphp: Slot [2] registered [ 0.270116] acpiphp: Slot [5] registered [ 0.271106] acpiphp: Slot [6] registered [ 0.273123] acpiphp: Slot [7] registered [ 0.274068] acpiphp: Slot [8] registered [ 0.275109] acpiphp: Slot [9] registered [ 0.277102] acpiphp: Slot [10] registered [ 0.278110] acpiphp: Slot [3] registered [ 0.280091] acpiphp: Slot [4] registered [ 0.281101] acpiphp: Slot [11] registered [ 0.283093] acpiphp: Slot [12] registered [ 0.285109] acpiphp: Slot [13] registered [ 0.286094] acpiphp: Slot [14] registered [ 0.287097] acpiphp: Slot [15] registered [ 0.289096] acpiphp: Slot [16] registered [ 0.290099] acpiphp: Slot [17] registered [ 0.292095] acpiphp: Slot [18] registered [ 0.294101] acpiphp: Slot [19] registered [ 0.295123] acpiphp: Slot [20] registered [ 0.297072] acpiphp: Slot [21] registered [ 0.298095] acpiphp: Slot [22] registered [ 0.300108] acpiphp: Slot [23] registered [ 0.302111] acpiphp: Slot [24] registered [ 0.304120] acpiphp: Slot [25] registered [ 0.306157] acpiphp: Slot [26] registered [ 0.308115] acpiphp: Slot [27] registered [ 0.310102] acpiphp: Slot [28] registered [ 0.311229] acpiphp: Slot [29] registered [ 0.313123] acpiphp: Slot [30] registered [ 0.315114] acpiphp: Slot [31] registered [ 0.317093] PCI host bridge to bus 0000:00 [ 0.318018] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.322022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.324019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.327024] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.331023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.334024] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.336173] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.340279] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.345010] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.355013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.359051] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.362016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.364014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.366016] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.367911] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.370963] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.373037] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.375716] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.381013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.391819] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.395795] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.402631] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.411015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.419017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.437014] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.447000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.454014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.461015] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.482017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.495630] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.503019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.511013] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.531961] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.543130] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.550017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.557015] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.575017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.584420] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.590014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.596017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.617016] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.624387] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.633013] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.639016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.655016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.667543] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.669443] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.672349] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.675168] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.678244] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.682077] iommu: Default domain type: Passthrough [ 0.684445] SCSI subsystem initialized [ 0.685096] ACPI: bus type USB registered [ 0.686094] usbcore: registered new interface driver usbfs [ 0.688068] usbcore: registered new interface driver hub [ 0.689080] usbcore: registered new device driver usb [ 0.690110] pps_core: LinuxPPS API ver. 1 registered [ 0.692008] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.694090] PTP clock support registered [ 0.696031] EDAC MC: Ver: 3.0.0 [ 0.697475] PCI: Using ACPI for IRQ routing [ 0.698878] NetLabel: Initializing [ 0.700012] NetLabel: domain hash size = 128 [ 0.702018] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.703074] NetLabel: unlabeled traffic allowed by default [ 0.705248] vgaarb: loaded [ 0.706272] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.708011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.714000] clocksource: Switched to clocksource kvm-clock [ 0.826270] VFS: Disk quotas dquot_6.6.0 [ 0.827659] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.830460] *** VALIDATE ramfs *** [ 0.831741] *** VALIDATE hugetlbfs *** [ 0.833977] pnp: PnP ACPI init [ 0.836527] pnp: PnP ACPI: found 6 devices [ 0.862489] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.865070] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.866669] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.868325] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.870211] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.872015] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.874118] NET: Registered protocol family 2 [ 0.876113] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.880423] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.883341] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.887553] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.890519] TCP: Hash tables configured (established 65536 bind 65536) [ 0.893236] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.896848] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.900070] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.903354] NET: Registered protocol family 1 [ 0.905775] RPC: Registered named UNIX socket transport module. [ 0.907782] RPC: Registered udp transport module. [ 0.909989] RPC: Registered tcp transport module. [ 0.911806] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.914258] NET: Registered protocol family 44 [ 0.915953] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.918264] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.920584] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.923050] PCI: CLS 0 bytes, default 64 [ 0.924874] Unpacking initramfs... [ 2.366891] debug: unmapping init [mem 0xffff93c33cc54000-0xffff93c33ffbffff] [ 2.371812] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.374799] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.377641] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.868556] Initialise system trusted keyrings [ 2.870092] Key type blacklist registered [ 2.872288] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.882363] zbud: loaded [ 2.885367] *** VALIDATE nfs *** [ 2.886740] *** VALIDATE nfs4 *** [ 2.888430] pstore: using deflate compression [ 2.892287] Platform Keyring initialized [ 2.990081] NET: Registered protocol family 38 [ 2.992081] Key type asymmetric registered [ 2.993340] Asymmetric key parser 'x509' registered [ 2.995181] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.997690] io scheduler mq-deadline registered [ 2.999184] io scheduler kyber registered [ 3.000626] io scheduler bfq registered [ 3.002302] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.004850] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.007607] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.010468] ACPI: Power Button [PWRF] [ 3.015762] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.022093] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.040210] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.047979] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.070729] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.099299] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.127681] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.133921] Non-volatile memory driver v1.3 [ 3.135293] Linux agpgart interface v0.103 [ 3.167299] virtio_blk virtio1: [vda] 149760 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.170713] vda: detected capacity change from 0 to 76677120 [ 3.190277] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.194435] vdb: detected capacity change from 0 to 1073741824 [ 3.211255] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.214379] vdc: detected capacity change from 0 to 2621440000 [ 3.232140] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.236454] vdd: detected capacity change from 0 to 2621440000 [ 3.253087] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.256129] vde: detected capacity change from 0 to 4294967296 [ 3.274653] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.278074] vdf: detected capacity change from 0 to 4294967296 [ 3.287288] libphy: Fixed MDIO Bus: probed [ 3.293469] usbcore: registered new interface driver usbserial_generic [ 3.296448] usbserial: USB Serial support registered for generic [ 3.299069] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.304613] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.306967] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.309734] mousedev: PS/2 mouse device common for all mice [ 3.313302] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.317237] rtc_cmos 00:05: RTC can wake from S4 [ 3.321541] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.322674] rtc_cmos 00:05: registered as rtc0 [ 3.329829] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.332717] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.332771] intel_pstate: CPU model not supported [ 3.336896] hid: raw HID events driver (C) Jiri Kosina [ 3.344899] usbcore: registered new interface driver usbhid [ 3.347054] usbhid: USB HID core driver [ 3.348652] drop_monitor: Initializing network drop monitor service [ 3.351172] Initializing XFRM netlink socket [ 3.353244] NET: Registered protocol family 10 [ 3.356361] Segment Routing with IPv6 [ 3.357985] NET: Registered protocol family 17 [ 3.360166] mpls_gso: MPLS GSO support [ 3.366786] RAS: Correctable Errors collector initialized. [ 3.369150] AVX version of gcm_enc/dec engaged. [ 3.370927] AES CTR mode by8 optimization enabled [ 3.443958] sched_clock: Marking stable (3443939073, 0)->(4372636964, -928697891) [ 3.447897] registered taskstats version 1 [ 3.449861] Loading compiled-in X.509 certificates [ 3.452208] zswap: loaded using pool lzo/zbud [ 3.476056] Key type big_key registered [ 3.487723] Key type encrypted registered [ 3.489646] ima: No TPM chip found, activating TPM-bypass! [ 3.492687] ima: Allocated hash algorithm: sha1 [ 3.495203] ima: No architecture policies found [ 3.497369] evm: Initialising EVM extended attributes: [ 3.499959] evm: security.selinux [ 3.501328] evm: security.ima [ 3.502535] evm: security.capability [ 3.504139] evm: HMAC attrs: 0x1 [ 3.506675] rtc_cmos 00:05: setting system clock to 2026-09-05 14:40:15 UTC (1788619215) [ 3.513107] debug: unmapping init [mem 0xffffffff8ca03000-0xffffffff8cbfffff] [ 3.517150] debug: unmapping init [mem 0xffffffff8b782000-0xffffffff8ba58fff] [ 3.527246] Write protecting the kernel read-only data: 28672k [ 3.530599] debug: unmapping init [mem 0xffffffff89e03000-0xffffffff89ffffff] [ 3.533624] debug: unmapping init [mem 0xffffffff8a714000-0xffffffff8a7fffff] [ 3.568210] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.577091] systemd[1]: Detected virtualization kvm. [ 3.579447] systemd[1]: Detected architecture x86-64. [ 3.581536] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.603952] systemd[1]: No hostname configured. [ 3.605749] systemd[1]: Set hostname to . [ 3.608129] random: systemd: uninitialized urandom read (16 bytes read) [ 3.610962] systemd[1]: Initializing machine ID from random generator. [ 3.657197] random: ln: uninitialized urandom read (6 bytes read) [ 3.735135] random: systemd: uninitialized urandom read (16 bytes read) [ 3.740102] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.745391] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.749907] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Reached target Paths. [ OK ] Reached target Slices. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.273073] random: fast init done [ 4.317036] device-mapper: uevent: version 1.0.3 [ 4.318848] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.983740] virtio_net virtio0 ens2: renamed from eth0 [ 5.040162] scsi host0: ata_piix [ 5.053191] scsi host1: ata_piix [ 5.054857] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.056218] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.817633] dracut-initqueue[580]: RTNETLINK answers: File exists [ 10.009625] random: crng init done [ 10.011155] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.397816] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.626153] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.922434] SELinux: Disabled at runtime. [ 11.990559] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.997985] systemd[1]: Detected virtualization kvm. [ 11.999540] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.550391] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.554389] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.559731] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.564669] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.567773] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.576086] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.580920] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-getty.slice. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting Kernel Debug File System... [ OK ] Created slice User and Session Slice. Starting udev Coldplug all Devices... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... Mounting Huge Pages File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ 12.763394] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.098681] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.417119] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.515946] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.626930] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.636410] EDAC sbridge: Ver: 1.1.2 [ 15.174792] Key type dns_resolver registered [ 15.493706] NFS: Registering the id_resolver key type [ 15.496428] Key type id_resolver registered [ 15.498090] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started Daily Cleanup of Temporary Directories. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started irqbalance daemon. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg309-server login: [ 43.584803] libcfs: loading out-of-tree module taints kernel. [ 43.605770] Key type ._llcrypt registered [ 43.607455] Key type .llcrypt registered [ 43.668458] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 64.603973] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 66.157849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 66.180856] alg: No test for adler32 (adler32-zlib) [ 66.521086] hrtimer: interrupt took 2623143 ns [ 67.836449] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 68.815684] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 70.775909] Key type lgssc registered [ 73.290572] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.481982] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 140.494231] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 155.376538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 155.428647] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.691570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 156.756434] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 156.928741] Lustre: lustre-MDT0000: new disk, initializing [ 157.069645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 157.094544] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 162.638726] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 177.247136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 177.385387] Lustre: 6506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 177.426942] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 177.436689] Lustre: Skipped 1 previous similar message [ 177.634455] Lustre: lustre-MDT0001: new disk, initializing [ 177.695610] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 177.735391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 177.747072] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 182.790870] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.966023] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 198.546740] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.072417] Lustre: lustre-OST0000: new disk, initializing [ 199.076561] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.081807] Lustre: 8444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.156859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 204.920824] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 206.871551] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 206.879225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 206.994564] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 219.644670] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 219.789749] Lustre: lustre-OST0001: new disk, initializing [ 219.796839] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 219.806405] Lustre: 9515:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 219.878788] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.792646] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.895856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.903581] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 228.993925] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 239.058761] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 249.890507] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 256.830768] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing check_logdir /tmp/testlogs/ [ 261.618161] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing yml_node [ 265.993122] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 268.621961] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 271.295583] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 273.173170] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 10:44:43 EDT 2026 [ 291.415583] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 293.208772] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 310.763284] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.764315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 310.769270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.788640] Lustre: Skipped 3 previous similar messages [ 315.875437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.886393] Lustre: Skipped 6 previous similar messages [ 316.877747] Lustre: server umount lustre-MDT0000 complete [ 320.703220] LustreError: 7919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619532 with bad export cookie 16489018987222951634 [ 320.721782] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 320.994409] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 321.026990] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 321.037555] Lustre: Skipped 1 previous similar message [ 326.115964] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.115976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.128226] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 326.142087] Lustre: Skipped 2 previous similar messages [ 330.208530] LustreError: 10149:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 330.209147] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.223902] LustreError: 10149:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 330.245476] Lustre: Skipped 1 previous similar message [ 335.327553] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 335.328058] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 335.363377] LustreError: 6512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 335.921692] Lustre: server umount lustre-MDT0001 complete [ 340.518455] Lustre: server umount lustre-OST0000 complete [ 345.171787] Lustre: server umount lustre-OST0001 complete [ 353.310254] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 365.095878] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 404.851559] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 405.208103] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 405.277861] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 405.409632] Lustre: lustre-MDT0000: new disk, initializing [ 405.506875] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.558116] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 409.649693] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 420.084367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 420.202576] Lustre: 15071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 420.266255] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 420.270104] Lustre: Skipped 1 previous similar message [ 420.354574] Lustre: lustre-MDT0001: new disk, initializing [ 420.463204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 420.509839] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 420.521723] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 425.475672] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 435.898249] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 442.950194] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 449.211737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 449.546611] Lustre: lustre-OST0000: new disk, initializing [ 449.552923] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 449.561911] Lustre: 17042:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 449.643241] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 450.900679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 450.908565] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 451.025854] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 455.316464] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 464.913562] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 470.565377] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 470.816341] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 475.219837] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 475.452932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 481.771867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 481.772231] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 481.774131] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.774137] Lustre: Skipped 2 previous similar messages [ 481.776034] LustreError: Skipped 1 previous similar message [ 481.825186] Lustre: Skipped 1 previous similar message [ 483.942941] Lustre: server umount lustre-OST0000 complete [ 492.002755] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 492.007811] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 492.044812] Lustre: Skipped 1 previous similar message [ 496.283114] Lustre: server umount lustre-MDT0000 complete [ 497.121365] LustreError: 15080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 497.138530] LustreError: 15080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 501.226307] LustreError: 15083:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.260599] LustreError: 15083:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 503.045783] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619715 with bad export cookie 16489018987222953167 [ 503.054320] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 503.078731] LustreError: 15064:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 503.832513] Lustre: server umount lustre-MDT0001 complete [ 509.204918] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:48:39 (1788619719) [ 512.608689] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 514.642715] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:48:44 (1788619724) [ 521.721514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 522.248526] LustreError: 19452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 522.368449] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 527.295632] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 536.508786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.467255] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 550.062666] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 556.099581] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 563.140274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 569.272753] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 576.693543] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 588.257501] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 588.273590] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 588.310123] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 588.318045] Lustre: Skipped 3 previous similar messages [ 593.438182] Lustre: server umount lustre-OST0000 complete [ 598.497535] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 598.509555] LustreError: Skipped 1 previous similar message [ 598.519112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 598.538648] Lustre: Skipped 1 previous similar message [ 611.295395] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 611.594900] Lustre: server umount lustre-MDT0000 complete [ 611.814403] LustreError: 19452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 611.827624] LustreError: 19452:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 616.000087] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788619827 with bad export cookie 16489018987222954049 [ 616.017706] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 616.041097] LustreError: 19434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 616.482822] Lustre: server umount lustre-MDT0001 complete [ 630.606491] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 633.658677] Key type lgssc unregistered [ 633.911378] LNet: 22986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 633.924369] LNetError: 22986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 634.999105] LNet: Removed LNI 192.168.203.109@tcp [ 636.085164] Key type .llcrypt unregistered [ 636.088300] Key type ._llcrypt unregistered [ 658.591243] Key type ._llcrypt registered [ 658.595843] Key type .llcrypt registered [ 658.712265] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 659.447696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 659.474371] alg: No test for adler32 (adler32-zlib) [ 660.494779] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 660.691909] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 662.352489] Key type lgssc registered [ 663.432469] Lustre: Echo OBD driver; http://www.lustre.org/ [ 672.781211] loop: module loaded [ 672.902694] loop0: detected capacity change from 0 to 204800000 [ 673.304518] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 673.323214] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 673.964058] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 680.351292] loop0: detected capacity change from 0 to 204800000 [ 680.714445] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 680.737844] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 680.781268] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.983981] loop0: detected capacity change from 0 to 204800000 [ 692.632543] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 692.709715] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 693.912880] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 693.959631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 694.030573] Lustre: lustre-MDT0000: new disk, initializing [ 694.175649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 694.213869] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 698.708727] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 707.698087] loop1: detected capacity change from 0 to 204800000 [ 712.408360] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 712.795310] Lustre: lustre-OST07c6: new disk, initializing [ 712.799388] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 712.803976] Lustre: Skipped 1 previous similar message [ 712.810470] Lustre: 26175:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 712.883194] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 718.643296] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 722.504568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 722.509672] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 722.549098] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 723.717595] Lustre: 26181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 249 < left 264, rollback = 2 [ 723.729415] Lustre: 26181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 723.734272] Lustre: 26181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 723.739153] Lustre: 26181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/2 [ 723.744230] Lustre: 26181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/5, delete: 0/0/0 [ 723.749352] Lustre: 26181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 739.576517] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 739.587697] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 739.597643] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 739.609630] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 739.615659] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 739.623643] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 744.397840] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 744.417841] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 744.423460] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 744.434066] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.444816] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 744.453144] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.461117] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 744.480472] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.494049] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 744.501242] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 744.506145] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 744.514639] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.700541] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 747.721247] Lustre: 25160:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 747.728315] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 747.734237] Lustre: 25160:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.740865] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 747.747948] Lustre: 25160:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.753374] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 747.759201] Lustre: 25160:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.766138] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 747.774061] Lustre: 25160:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 747.780612] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 747.786066] Lustre: 25160:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.598666] Lustre: 26181:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 752.606225] Lustre: 26181:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 752.617515] Lustre: 26181:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 752.625582] Lustre: 26181:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.631435] Lustre: 26181:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 752.636354] Lustre: 26181:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.640567] Lustre: 26181:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 752.645543] Lustre: 26181:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.654588] Lustre: 26181:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 752.659459] Lustre: 26181:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 752.666960] Lustre: 26181:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 752.673445] Lustre: 26181:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 763.715905] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 763.726154] Lustre: 25158:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 763.731965] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.738293] Lustre: 25158:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 763.744436] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.749701] Lustre: 25158:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 763.755566] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 763.763160] Lustre: 25158:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 763.768538] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 763.774639] Lustre: 25158:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 763.780415] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 763.785675] Lustre: 25158:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 814.563791] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 814.577134] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 814.594141] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 819.691823] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 824.801274] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 826.335161] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 826.466463] Lustre: server umount lustre-OST07c6 complete [ 835.439683] Lustre: server umount lustre-MDT0000 complete [ 850.545753] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 853.355529] Key type lgssc unregistered [ 853.682610] LNet: 28729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 853.689382] LNetError: 28729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.761370] LNet: Removed LNI 192.168.203.109@tcp [ 855.634746] Key type .llcrypt unregistered [ 855.637241] Key type ._llcrypt unregistered [ 873.872686] Key type ._llcrypt registered [ 873.875152] Key type .llcrypt registered [ 873.999300] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 874.937978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 874.987581] alg: No test for adler32 (adler32-zlib) [ 876.228418] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 876.466306] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 878.167275] Key type lgssc registered [ 879.294361] Lustre: Echo OBD driver; http://www.lustre.org/ [ 887.483310] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 887.498823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.952371] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 893.045675] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 898.541472] LustreError: 29996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 898.561723] LustreError: 29996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 902.537922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 903.049154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 907.782366] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 915.792493] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 920.886141] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 927.462546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 927.666022] LustreError: 31817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 927.946061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 933.140761] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 934.130105] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 942.625473] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 964.064519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 964.072458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 964.080594] LustreError: Skipped 1 previous similar message [ 964.094630] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 964.095589] Lustre: Skipped 1 previous similar message [ 964.126634] Lustre: Skipped 1 previous similar message [ 966.196660] Lustre: server umount lustre-OST0000 complete [ 974.311263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 974.324978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 974.326605] Lustre: Skipped 1 previous similar message [ 974.338443] Lustre: Skipped 1 previous similar message [ 977.212479] Lustre: server umount lustre-MDT0000 complete [ 979.429995] LustreError: 29996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 979.468358] LustreError: 29996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 980.986568] LustreError: 32984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620192 with bad export cookie 10384190237899973329 [ 980.990982] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 980.997117] LustreError: 32984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 981.454936] Lustre: server umount lustre-MDT0001 complete [ 994.136986] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 997.597786] Key type lgssc unregistered [ 997.934026] LNet: 33736:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 997.944503] LNetError: 33736:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.023655] LNet: Removed LNI 192.168.203.109@tcp [ 999.875292] Key type .llcrypt unregistered [ 999.878463] Key type ._llcrypt unregistered [ 1021.707929] Key type ._llcrypt registered [ 1021.718062] Key type .llcrypt registered [ 1021.807784] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1022.466244] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1022.477775] alg: No test for adler32 (adler32-zlib) [ 1023.677310] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 1024.038450] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1025.711251] Key type lgssc registered [ 1027.531862] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1037.035112] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1037.060718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1038.556503] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1043.297065] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1048.033327] LustreError: 35005:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1048.046701] LustreError: 35005:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1051.525651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1051.986414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1057.087862] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1064.636582] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1069.950310] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1077.312507] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1077.787299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1083.961592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1084.731147] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1093.445757] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1109.485533] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1109.501066] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.516246] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1111.521028] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1111.533984] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1111.535067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1111.561971] Lustre: Skipped 1 previous similar message [ 1113.272038] Lustre: server umount lustre-OST0000 complete [ 1118.692277] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1118.703836] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1118.732051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.762358] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1121.764694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1121.797903] Lustre: Skipped 1 previous similar message [ 1123.925974] Lustre: server umount lustre-MDT0000 complete [ 1126.891977] LustreError: 35001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1126.936976] LustreError: 35001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1128.298831] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620340 with bad export cookie 6918270261601576801 [ 1128.303962] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1128.320556] LustreError: 34983:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1128.757801] Lustre: server umount lustre-MDT0001 complete [ 1140.557259] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1143.722511] Key type lgssc unregistered [ 1144.062772] LNet: 38559:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1144.069886] LNetError: 38559:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1144.095043] LNet: Removed LNI 192.168.203.109@tcp [ 1145.070578] Key type .llcrypt unregistered [ 1145.075533] Key type ._llcrypt unregistered [ 1171.183901] Key type ._llcrypt registered [ 1171.186529] Key type .llcrypt registered [ 1171.294936] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1172.595057] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1172.649349] alg: No test for adler32 (adler32-zlib) [ 1173.793320] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 1174.143733] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1175.823110] Key type lgssc registered [ 1177.237594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1184.696579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1184.733985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1186.268593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1190.609939] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1196.514929] LustreError: 39974:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.535139] LustreError: 39974:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1199.584527] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1200.107426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1205.833834] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1215.952197] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1222.414931] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1228.668596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1228.902614] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1232.040086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1235.062410] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1242.459987] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1254.781263] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) [ 1255.912977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1255.921236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1255.938389] Lustre: Skipped 2 previous similar messages [ 1255.953852] Lustre: Skipped 1 previous similar message [ 1258.215911] Lustre: server umount lustre-MDT0000 complete [ 1259.870593] LustreError: 39969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.024269] LustreError: 39973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1261.062439] LustreError: 39973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1262.172309] LustreError: 39955:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620474 with bad export cookie 18222163648408828128 [ 1262.190407] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1264.986136] LustreError: 39969:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1265.024555] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.9@tcp (stopping) [ 1265.033566] Lustre: Skipped 1 previous similar message [ 1266.153640] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1267.177956] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1267.194359] Lustre: Skipped 1 previous similar message [ 1268.817966] Lustre: server umount lustre-MDT0001 complete [ 1283.797069] Lustre: server umount lustre-OST0000 complete [ 1300.172744] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1302.873558] Key type lgssc unregistered [ 1303.185452] LNet: 43844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1303.195394] LNetError: 43844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1303.223610] LNet: Removed LNI 192.168.203.109@tcp [ 1304.107567] Key type .llcrypt unregistered [ 1304.110799] Key type ._llcrypt unregistered [ 1327.229127] Key type ._llcrypt registered [ 1327.233063] Key type .llcrypt registered [ 1327.319134] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1328.620092] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1328.708426] alg: No test for adler32 (adler32-zlib) [ 1329.786143] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 1330.041222] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1331.719181] Key type lgssc registered [ 1332.803341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1340.461281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1340.489871] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1342.054271] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1346.736724] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1351.141913] LustreError: 45258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1351.163188] LustreError: 45258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1356.101342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1356.257949] LustreError: 45257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1356.507269] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1361.536582] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1369.774683] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1375.992340] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1383.680948] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1384.144505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1387.258788] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1390.790882] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1398.456894] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1407.977105] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1407.984240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1407.999313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1408.008981] LustreError: Skipped 1 previous similar message [ 1408.024226] Lustre: Skipped 1 previous similar message [ 1409.682615] Lustre: server umount lustre-OST0000 complete [ 1428.449899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1428.452693] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.478207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1428.491970] Lustre: Skipped 2 previous similar messages [ 1429.537454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1429.545756] Lustre: Skipped 2 previous similar messages [ 1434.593909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1434.598361] Lustre: Skipped 1 previous similar message [ 1439.714963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1439.729819] Lustre: Skipped 1 previous similar message [ 1442.271321] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1442.578093] Lustre: server umount lustre-MDT0000 complete [ 1444.834073] LustreError: 45257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1444.857101] LustreError: 45257:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1446.237678] LustreError: 45239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620658 with bad export cookie 1744687846663519389 [ 1446.251267] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1446.256172] LustreError: 45239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1452.687512] Lustre: server umount lustre-MDT0001 complete [ 1463.050416] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1465.741019] Key type lgssc unregistered [ 1466.004574] LNet: 48967:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1466.011422] LNetError: 48967:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1466.024821] LNet: Removed LNI 192.168.203.109@tcp [ 1466.939218] Key type .llcrypt unregistered [ 1466.941240] Key type ._llcrypt unregistered [ 1488.002294] Key type ._llcrypt registered [ 1488.004438] Key type .llcrypt registered [ 1488.084816] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1489.147814] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1489.194695] alg: No test for adler32 (adler32-zlib) [ 1490.255861] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 1490.458068] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1492.119399] Key type lgssc registered [ 1493.135914] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1500.661873] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1500.672900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1502.196657] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1507.264552] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1512.423696] LustreError: 50380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1512.455847] LustreError: 50380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1516.731682] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1517.262496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1521.907740] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1529.615529] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1536.099460] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1542.646634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1542.986642] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1547.068283] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1549.538646] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1557.390970] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1566.515711] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1570.828545] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1583.436629] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0511h06m33s [ 1585.393426] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1588.200486] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.203587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1588.217993] Lustre: Skipped 1 previous similar message [ 1589.282400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1589.295713] Lustre: Skipped 4 previous similar messages [ 1591.208900] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) [ 1591.222787] Lustre: Skipped 1 previous similar message [ 1593.957764] Lustre: server umount lustre-MDT0000 complete [ 1595.371177] LustreError: 51621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.417846] LustreError: 51621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1597.270443] LustreError: 50375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1597.621537] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1601.510296] LustreError: 51621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1601.542210] LustreError: 51621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1605.772923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.925414] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1606.230532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1606.309328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1607.278566] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1611.423876] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1615.728943] Lustre: DEBUG MARKER: Wait for df (36696) ... [ 1617.194850] Lustre: DEBUG MARKER: done [ 1623.015491] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1623.034447] Lustre: Skipped 1 previous similar message [ 1623.045475] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1628.135300] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1628.143277] Lustre: Skipped 2 previous similar messages [ 1636.320863] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1636.706970] Lustre: server umount lustre-OST0000 complete [ 1643.488777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1643.494253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.498051] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1643.506933] Lustre: Skipped 2 previous similar messages [ 1643.518682] Lustre: Skipped 3 previous similar messages [ 1647.508365] Lustre: server umount lustre-MDT0000 complete [ 1650.656404] LustreError: 51108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.677240] LustreError: 51108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1652.382145] LustreError: 50361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788620864 with bad export cookie 3918584083022504018 [ 1652.384418] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1652.394466] LustreError: 50361:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1653.004301] Lustre: server umount lustre-MDT0001 complete [ 1666.182692] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1669.525921] Key type lgssc unregistered [ 1669.755133] LNet: 55307:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.758802] LNetError: 55307:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1669.772306] LNet: Removed LNI 192.168.203.109@tcp [ 1670.476699] Key type .llcrypt unregistered [ 1670.478440] Key type ._llcrypt unregistered [ 1713.345774] Key type ._llcrypt registered [ 1713.349422] Key type .llcrypt registered [ 1713.502715] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1715.088811] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1715.125201] alg: No test for adler32 (adler32-zlib) [ 1716.284869] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 1716.582826] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1718.311125] Key type lgssc registered [ 1719.715141] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1730.031649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1730.077206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1731.731982] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1731.748507] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1732.020516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.047318] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1746.551892] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.757326] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1746.780917] Lustre: 57510:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1747.089266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1752.084496] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1761.102092] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1767.396475] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1775.441737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1775.751596] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1776.021114] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1783.447760] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1786.051485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1791.928970] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1800.017039] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0511h10m09s [ 1802.141496] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1810.343618] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1813.077887] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1813.156448] LustreError: 58970:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1833.425521] Lustre: lustre-MDT0000: Client e7a2b9d5-5f1f-4c4c-9e96-19be7e16bc3b (at 192.168.203.9@tcp) reconnecting [ 1853.183092] LustreError: 58970:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1855.747454] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1858.930822] Lustre: DEBUG MARKER: done [ 1867.744551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1867.766355] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.789318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1868.275544] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1872.457748] Lustre: server umount lustre-OST0000 complete [ 1878.498134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1878.500348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1878.522929] Lustre: Skipped 2 previous similar messages [ 1878.532272] Lustre: Skipped 2 previous similar messages [ 1879.589760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1879.600772] Lustre: Skipped 2 previous similar messages [ 1882.969891] Lustre: server umount lustre-MDT0000 complete [ 1887.090596] LustreError: 57503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621099 with bad export cookie 2598351355561330417 [ 1887.096953] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1887.110975] LustreError: 57503:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1887.724982] Lustre: server umount lustre-MDT0001 complete [ 1900.223380] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 1903.489220] Key type lgssc unregistered [ 1903.851363] LNet: 61838:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1903.862015] LNetError: 61838:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1903.909698] LNet: Removed LNI 192.168.203.109@tcp [ 1904.873361] Key type .llcrypt unregistered [ 1904.876072] Key type ._llcrypt unregistered [ 1952.106371] Key type ._llcrypt registered [ 1952.109870] Key type .llcrypt registered [ 1952.210706] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 1953.201381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1953.252086] alg: No test for adler32 (adler32-zlib) [ 1954.480733] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 1954.803096] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 1956.495178] Key type lgssc registered [ 1957.560304] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1961.257267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1961.283486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1962.519685] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1962.539279] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1962.674792] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1963.765332] Lustre: Failing over lustre-MDT0000 [ 1964.009591] Lustre: server umount lustre-MDT0000 complete [ 1970.889042] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:13:01 (1788621181) [ 1978.201228] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1978.828951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1983.801348] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 1992.956625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1993.078604] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1993.118714] Lustre: 64632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1993.453534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1998.712511] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2007.432930] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2015.231233] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2025.565265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2025.791786] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2026.139146] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2033.202001] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2034.861077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2041.799648] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2047.579629] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2054.059896] LustreError: 63556:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000009bbac346 x1875505227798016/t0(0) o2->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 440/432 e 0 to 0 dl 0 ref 1 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2055.140281] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2055.151140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.166186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.654227] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2055.667137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2055.674275] Lustre: Skipped 1 previous similar message [ 2060.360989] Lustre: server umount lustre-MDT0000 complete [ 2063.776819] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621275 with bad export cookie 5232765271171915402 [ 2063.782970] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2063.789078] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2064.080831] Lustre: server umount lustre-MDT0001 complete [ 2067.613912] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2071.979072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2080.033050] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2081.247269] Lustre: 63555:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788621277/real 1788621277] req@00000000ae6052ad x1875505227801216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788621293 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2081.270417] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2081.277732] Lustre: Skipped 1 previous similar message [ 2084.383151] Lustre: 63555:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788621280/real 1788621280] req@00000000de8c8894 x1875505227801344/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788621296 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2086.367162] Lustre: 63554:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788621282/real 1788621282] req@00000000725c7886 x1875505227801472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788621298 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2089.922736] LustreError: 66481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2089.942618] LustreError: 66481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2090.018969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2090.113305] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2091.039175] Lustre: 63554:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788621286/real 1788621286] req@00000000d8f44ab4 x1875505227801728/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788621302 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2091.049039] LustreError: 66482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.079188] LustreError: 66482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2094.201848] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2095.137303] Lustre: 63556:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788621291/real 1788621291] req@000000005e0df3d0 x1875505227801856/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788621307 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2096.099711] LustreError: 66481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2096.123891] LustreError: 66481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2101.225389] LustreError: 66482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2101.250154] LustreError: 66482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2103.629022] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2104.011390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2108.416559] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2109.436959] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2110.510972] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2116.690904] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2123.336512] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2130.922324] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2130.925454] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2130.934908] Lustre: Skipped 2 previous similar messages [ 2130.947273] Lustre: Skipped 2 previous similar messages [ 2136.039385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2136.053599] Lustre: Skipped 2 previous similar messages [ 2136.991424] Lustre: server umount lustre-MDT0000 complete [ 2140.853125] LustreError: 64624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621352 with bad export cookie 5232765271171917677 [ 2140.861216] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2140.874442] LustreError: 64624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2141.153322] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2141.155925] LustreError: 68401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2141.197606] LustreError: 68401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2141.304238] Lustre: server umount lustre-MDT0001 complete [ 2148.012828] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2157.883241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2165.678228] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x489e844a2d33ecd0 [ 2165.703518] Lustre: MGC192.168.203.109@tcp: Connection restored to 0@lo (at 0@lo) [ 2166.001861] LustreError: 66481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2166.066760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2166.162851] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2170.156178] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2170.502310] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2178.916131] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2180.398467] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2184.002994] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2191.720204] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2197.440949] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2205.153563] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2205.166915] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2205.193759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2210.811588] Lustre: server umount lustre-MDT0000 complete [ 2211.303889] LustreError: 71082:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2211.321300] LustreError: 71082:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2214.599213] LustreError: 64623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621426 with bad export cookie 5232765271171919056 [ 2214.608537] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2214.614383] LustreError: 64623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2214.859732] Lustre: server umount lustre-MDT0001 complete [ 2221.492332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2225.126749] Lustre: DEBUG MARKER: files compared the same [ 2237.748786] Lustre: server umount lustre-OST0000 complete [ 2254.984771] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2257.969728] Key type lgssc unregistered [ 2258.310643] LNet: 74360:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2258.320083] LNetError: 74360:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.337766] LNet: Removed LNI 192.168.203.109@tcp [ 2259.319140] Key type .llcrypt unregistered [ 2259.326745] Key type ._llcrypt unregistered [ 2283.315568] Key type ._llcrypt registered [ 2283.318585] Key type .llcrypt registered [ 2283.414557] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2284.508969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2284.545784] alg: No test for adler32 (adler32-zlib) [ 2285.553399] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 2285.701283] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2287.367224] Key type lgssc registered [ 2288.437788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2297.037578] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2297.066156] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2298.647521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2303.357236] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug +malloc all [ 2308.068868] LustreError: 75774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.090280] LustreError: 75774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2312.938647] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.168647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2317.771724] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug +malloc all [ 2328.053732] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2334.636630] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2344.391715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2344.678888] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2350.426459] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug +malloc all [ 2358.145939] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2368.241568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2378.723335] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2378.739114] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2378.756382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2378.767285] Lustre: Skipped 1 previous similar message [ 2383.851846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2383.858157] Lustre: Skipped 1 previous similar message [ 2384.200263] Lustre: server umount lustre-OST0000 complete [ 2388.970316] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2388.996766] Lustre: Skipped 2 previous similar messages [ 2388.997714] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2389.014973] Lustre: Skipped 1 previous similar message [ 2394.081402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2394.095771] Lustre: Skipped 1 previous similar message [ 2394.513183] Lustre: server umount lustre-MDT0000 complete [ 2398.123496] LustreError: 75754:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621610 with bad export cookie 2070319102672196551 [ 2398.137769] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2398.466550] Lustre: server umount lustre-MDT0001 complete [ 2410.939293] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2414.021786] Key type lgssc unregistered [ 2414.319626] LNet: 79330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2414.330958] LNetError: 79330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2414.364295] LNet: Removed LNI 192.168.203.109@tcp [ 2415.253374] Key type .llcrypt unregistered [ 2415.255937] Key type ._llcrypt unregistered [ 2439.765944] Key type ._llcrypt registered [ 2439.768313] Key type .llcrypt registered [ 2439.873753] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2440.989480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2441.095453] alg: No test for adler32 (adler32-zlib) [ 2442.231805] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 2442.586326] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2444.288080] Key type lgssc registered [ 2445.327729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2453.495755] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2453.520175] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2469.087370] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2474.207310] LustreError: 80715:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [ 2500.834803] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2509.285862] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2519.478173] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2528.628335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2528.855277] Lustre: *** cfs_fail_loc=706, val=0*** [ 2529.346684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2529.431366] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2534.371904] LustreError: 80741:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2534.391928] LustreError: 80741:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2534.406482] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2535.401431] LustreError: 81803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2535.425586] LustreError: 81803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2540.513612] LustreError: 81802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2540.536243] LustreError: 81802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2542.792327] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2543.113311] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2547.526078] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2555.317755] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2561.626800] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2565.088393] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2565.092922] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2565.114815] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2568.679017] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2568.680416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2573.794403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2573.816813] Lustre: Skipped 2 previous similar messages [ 2578.399191] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2578.582607] Lustre: server umount lustre-OST0000 complete [ 2584.034960] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2584.052892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2584.060636] Lustre: Skipped 1 previous similar message [ 2588.820447] Lustre: server umount lustre-MDT0000 complete [ 2589.153162] LustreError: 81803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.165316] LustreError: 81803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2592.892914] LustreError: 83378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788621804 with bad export cookie 17814194832671407049 [ 2592.904252] LustreError: 83378:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2592.911526] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2593.359119] Lustre: server umount lustre-MDT0001 complete [ 2604.781972] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2607.650259] Key type lgssc unregistered [ 2607.945350] LNet: 84346:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2607.954477] LNetError: 84346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2609.000270] LNet: Removed LNI 192.168.203.109@tcp [ 2610.015162] Key type .llcrypt unregistered [ 2610.017912] Key type ._llcrypt unregistered [ 2635.253120] Key type ._llcrypt registered [ 2635.260371] Key type .llcrypt registered [ 2635.365047] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2636.695726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2636.729710] alg: No test for adler32 (adler32-zlib) [ 2637.823959] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 2638.103634] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 2639.783169] Key type lgssc registered [ 2640.897969] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2648.144938] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2648.164568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2654.079340] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2662.684330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2663.276428] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2667.815949] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2668.514897] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2668.542164] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2673.635139] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2673.665789] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2675.425899] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2675.855931] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2675.876875] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2675.894496] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2681.318291] LustreError: 86446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2681.342657] LustreError: 86446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2682.479836] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2686.434575] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2686.461953] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2694.167840] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2694.190943] Lustre: MGS: Client 8f8044cc-6c27-4731-b79a-ce0a583067bd (at 0@lo) reconnecting [ 2694.252062] LustreError: 86461:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2694.567472] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2694.655378] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2699.927898] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2716.640734] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2716.649150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2716.662460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2720.227803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2720.232772] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2722.223321] Lustre: server umount lustre-OST0000 complete [ 2730.473189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2730.511513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2730.525318] Lustre: Skipped 2 previous similar messages [ 2732.453172] Lustre: server umount lustre-MDT0000 complete [ 2735.586108] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2735.613855] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 2737.151647] Lustre: server umount lustre-MDT0001 complete [ 2741.443845] Lustre: server umount MGS complete [ 2754.351639] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:26:04 (1788621964) [ 2785.233626] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 2795.447399] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 2834.988778] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2839.999467] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2851.035799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2851.326216] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2851.374352] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2851.528207] Lustre: lustre-MDT0001: new disk, initializing [ 2851.671114] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2851.714202] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2851.721393] Lustre: 93647:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2851.727705] LustreError: 93647:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2856.620684] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2868.393058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2868.757876] Lustre: lustre-OST0000: new disk, initializing [ 2868.764670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2868.773719] Lustre: 94412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2868.891541] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2875.507882] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2876.455649] Lustre: 94751:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000067170e8e x1875505945028736/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788622104 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2886.504580] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2893.703223] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2893.718390] Lustre: MGS: Client ccf04a9e-96c6-43fe-8c6d-eb16d740ec9b (at 0@lo) reconnecting [ 2893.766126] Lustre: 92913:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2893.777793] LustreError: 93633:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2893.802299] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2893.914833] Lustre: lustre-MDT0000: new disk, initializing [ 2894.095801] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2894.120409] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2894.136077] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2895.203867] Lustre: 95481:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000083bbfccc x1875505945035904/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788622123 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2898.984669] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 2900.302218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2900.321833] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2900.459765] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2924.512441] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2924.523311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2924.534563] Lustre: Skipped 1 previous similar message [ 2924.538463] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2926.054720] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2931.198041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2931.211977] Lustre: Skipped 2 previous similar messages [ 2938.337615] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2938.555419] Lustre: server umount lustre-OST0000 complete [ 2946.535258] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2946.536794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2946.559528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2946.572256] Lustre: Skipped 2 previous similar messages [ 2946.586601] Lustre: Skipped 4 previous similar messages [ 2948.787054] Lustre: server umount lustre-MDT0000 complete [ 2951.650774] LustreError: 93622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2951.680948] LustreError: 93622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2959.143632] Lustre: server umount lustre-MDT0001 complete [ 2963.099762] Lustre: server umount MGS complete [ 2974.602955] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:29:45 (1788622185) [ 2994.173831] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 2997.244621] Key type lgssc unregistered [ 2997.558747] LNet: 97982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2997.584131] LNetError: 97982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2997.607184] LNet: Removed LNI 192.168.203.109@tcp [ 2998.592588] Key type .llcrypt unregistered [ 2998.620169] Key type ._llcrypt unregistered [ 3016.732121] Key type ._llcrypt registered [ 3016.733471] Key type .llcrypt registered [ 3016.878485] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3018.314382] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3018.346065] alg: No test for adler32 (adler32-zlib) [ 3019.627310] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 3019.951624] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 3021.623157] Key type lgssc registered [ 3022.843941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3030.667209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3030.706189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3032.249257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3036.605564] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3042.786901] LustreError: 99200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3042.813986] LustreError: 99200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3044.900864] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3045.173315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3049.584033] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3057.234612] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3063.163895] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3070.120297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3070.464057] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3076.980749] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3077.612120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3084.436841] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3092.962066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3092.967820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3092.971518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3092.993304] Lustre: Skipped 1 previous similar message [ 3098.087789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3098.096635] Lustre: Skipped 1 previous similar message [ 3098.324922] Lustre: server umount lustre-OST0000 complete [ 3103.206314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3103.206972] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3103.220609] Lustre: Skipped 1 previous similar message [ 3103.226098] Lustre: Skipped 2 previous similar messages [ 3108.228566] Lustre: server umount lustre-MDT0000 complete [ 3108.321167] LustreError: 99200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3108.341823] LustreError: 99200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3112.555402] LustreError: 99180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622324 with bad export cookie 14330390456051127852 [ 3112.558113] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3112.571223] LustreError: 99180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3112.901165] Lustre: server umount lustre-MDT0001 complete [ 3125.273548] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 3128.031746] Key type lgssc unregistered [ 3128.393121] LNet: 102729:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3128.415741] LNetError: 102729:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3128.437858] LNet: Removed LNI 192.168.203.109@tcp [ 3129.217105] Key type .llcrypt unregistered [ 3129.218792] Key type ._llcrypt unregistered [ 3145.093744] Key type ._llcrypt registered [ 3145.101190] Key type .llcrypt registered [ 3145.201476] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3146.049340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3146.104823] alg: No test for adler32 (adler32-zlib) [ 3147.298308] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 3147.468175] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 3149.121109] Key type lgssc registered [ 3150.328419] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3156.968497] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3156.972656] LustreError: 103823:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3156.974525] Lustre: Skipped 1 previous similar message [ 3156.974541] LustreError: 103824:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3156.994574] LustreError: 103823:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3157.004176] LustreError: 103823:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3157.007675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3157.012060] LustreError: 103823:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3158.444631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3166.231152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3166.550705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3170.997600] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3176.967740] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing lsmod [ 3180.057374] LustreError: 105030:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3180.073434] LustreError: 105031:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3180.081326] LustreError: 105030:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4984 [ 3180.089104] LustreError: 105030:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3180.093533] LustreError: 105030:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3180.114404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3180.533604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3183.604925] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3187.171372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3187.184227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3187.206505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.706435] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3188.711436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3188.737679] Lustre: Skipped 1 previous similar message [ 3188.742964] Lustre: Skipped 1 previous similar message [ 3190.516745] Lustre: server umount lustre-MDT0000 complete [ 3191.851624] LustreError: 103857:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3191.871517] LustreError: 103857:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3194.412632] LustreError: 103836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622406 with bad export cookie 8715353100706099670 [ 3194.424217] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3195.999625] LustreError: 104292:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000cb470f15 x1875506478732288/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3196.202950] Lustre: server umount lustre-MDT0001 complete [ 3211.253485] Lustre: server umount lustre-OST0000 complete [ 3220.452936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3221.102279] LustreError: 106096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3221.211747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3226.087932] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3226.600251] LustreError: 106097:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3231.716676] LustreError: 106096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3235.419680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3235.834058] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3240.438306] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3248.661846] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3255.137971] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3262.565080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3263.046330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3266.099757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 3269.463677] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3277.391576] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3290.086104] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3290.103627] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3290.118038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3290.132625] Lustre: Skipped 1 previous similar message [ 3295.721410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3295.745640] Lustre: Skipped 2 previous similar messages [ 3296.434949] Lustre: server umount lustre-OST0000 complete [ 3300.836701] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3300.842532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3300.853661] Lustre: Skipped 2 previous similar messages [ 3300.871465] Lustre: Skipped 1 previous similar message [ 3306.638228] Lustre: server umount lustre-MDT0000 complete [ 3311.074826] LustreError: 106096:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3311.103788] LustreError: 106096:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3311.431271] LustreError: 106078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622523 with bad export cookie 8715353100706100601 [ 3311.432822] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3311.444384] LustreError: 106078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 3311.928222] Lustre: server umount lustre-MDT0001 complete [ 3324.429978] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 3327.020219] Key type lgssc unregistered [ 3327.295122] LNet: 109653:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3327.307208] LNetError: 109653:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3327.319922] LNet: Removed LNI 192.168.203.109@tcp [ 3328.158497] Key type .llcrypt unregistered [ 3328.161266] Key type ._llcrypt unregistered [ 3351.304078] Key type ._llcrypt registered [ 3351.306049] Key type .llcrypt registered [ 3351.430360] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3352.546824] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3352.680544] alg: No test for adler32 (adler32-zlib) [ 3353.761438] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 3354.033344] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 3355.704077] Key type lgssc registered [ 3357.056242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3365.995811] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3366.015327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3367.496762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3372.239189] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3377.638689] LustreError: 111065:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3377.669017] LustreError: 111065:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3381.558038] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3381.911709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3386.546786] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3394.526434] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3401.114812] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3408.846640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3409.192103] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.284123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3415.767519] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3423.993813] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3432.731075] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3437.424872] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3443.173220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3443.180491] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3443.199294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3447.265387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3447.268021] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3447.290326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3448.571066] Lustre: server umount lustre-OST0000 complete [ 3453.920383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3453.927422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3453.937647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3453.944409] Lustre: Skipped 1 previous similar message [ 3457.505776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3457.515792] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3462.624434] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3462.638311] Lustre: Skipped 2 previous similar messages [ 3467.231526] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3467.636544] Lustre: server umount lustre-MDT0000 complete [ 3467.760869] LustreError: 111066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3467.780619] LustreError: 111066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3471.737834] LustreError: 111045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622683 with bad export cookie 4599236391642115540 [ 3471.763958] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3472.280798] Lustre: server umount lustre-MDT0001 complete [ 3485.913597] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 3489.104751] Key type lgssc unregistered [ 3489.462206] LNet: 114705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3489.485666] LNetError: 114705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3489.515505] LNet: Removed LNI 192.168.203.109@tcp [ 3490.597349] Key type .llcrypt unregistered [ 3490.599323] Key type ._llcrypt unregistered [ 3510.738275] Key type ._llcrypt registered [ 3510.742848] Key type .llcrypt registered [ 3510.937884] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3512.004654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3512.031448] alg: No test for adler32 (adler32-zlib) [ 3513.283944] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 3513.596613] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 3515.440180] Key type lgssc registered [ 3517.358824] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3526.805926] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3526.831366] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3528.210196] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3532.470205] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3538.408520] LustreError: 115924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3538.421345] LustreError: 115924:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3540.917417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3541.417990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3546.020587] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3554.341736] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3559.527330] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3566.093132] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3566.443849] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.234861] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3573.641162] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3581.213402] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3593.199530] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3593.211379] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.227640] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3594.215845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3594.221490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3594.229201] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3595.741698] Lustre: server umount lustre-OST0000 complete [ 3602.912263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3602.920666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.935054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3602.941182] Lustre: Skipped 1 previous similar message [ 3606.387408] Lustre: server umount lustre-MDT0000 complete [ 3609.570680] LustreError: 115919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3609.605333] LustreError: 115919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3610.351447] LustreError: 115904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788622822 with bad export cookie 14035134009368872422 [ 3610.361740] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3610.982478] Lustre: server umount lustre-MDT0001 complete [ 3622.692310] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 3625.650578] Key type lgssc unregistered [ 3625.979980] LNet: 119456:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3625.983339] LNetError: 119456:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3626.000692] LNet: Removed LNI 192.168.203.109@tcp [ 3626.848156] Key type .llcrypt unregistered [ 3626.850464] Key type ._llcrypt unregistered [ 3653.585473] Key type ._llcrypt registered [ 3653.588307] Key type .llcrypt registered [ 3653.682971] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3654.357799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3654.488217] alg: No test for adler32 (adler32-zlib) [ 3655.649523] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 3656.020650] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 3657.791590] Key type lgssc registered [ 3658.911445] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3668.611088] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3668.640638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3670.108199] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3674.800338] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3680.227275] LustreError: 120871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3680.250763] LustreError: 120871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3683.993160] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3684.451606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3688.461703] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3696.008404] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3702.621977] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3709.433771] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3709.853043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3716.012392] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 3716.188105] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3723.396409] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3733.420875] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3736.917805] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3736.921222] Lustre: Skipped 2 previous similar messages [ 3740.775960] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3740.851401] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3741.550729] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3741.557140] Lustre: Skipped 2 previous similar messages [ 3749.436733] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3749.442288] Lustre: Skipped 1 previous similar message [ 3753.858668] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.9@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3753.871495] Lustre: Skipped 2 previous similar messages [ 3759.691032] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.9@tcp 192.168.0.[2,10]@tcp [ 3762.851349] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.9/24@tcp in log lustre-MDT0000 [ 3762.858263] Lustre: Skipped 2 previous similar messages [ 3768.382433] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.9/24@tcp [ 3768.394093] Lustre: Skipped 1 previous similar message [ 3775.459633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3775.464567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3775.471110] Lustre: Skipped 1 previous similar message [ 3775.472522] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3775.485300] Lustre: Skipped 1 previous similar message [ 3780.470972] Lustre: server umount lustre-OST0000 complete [ 3785.697591] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3785.708093] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3785.744673] Lustre: Skipped 1 previous similar message [ 3785.758475] Lustre: Skipped 1 previous similar message [ 3790.829446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3790.834719] Lustre: Skipped 1 previous similar message [ 3790.921704] Lustre: server umount lustre-MDT0000 complete [ 3794.218692] LustreError: 120852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623006 with bad export cookie 9961642943677211406 [ 3794.221314] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3794.232247] LustreError: 120852:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3794.589409] Lustre: server umount lustre-MDT0001 complete [ 3805.601587] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 3808.476106] Key type lgssc unregistered [ 3808.808967] LNet: 125912:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3808.818230] LNetError: 125912:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3808.836986] LNet: Removed LNI 192.168.203.109@tcp [ 3809.560985] Key type .llcrypt unregistered [ 3809.563323] Key type ._llcrypt unregistered [ 3841.627832] Key type ._llcrypt registered [ 3841.632430] Key type .llcrypt registered [ 3841.736231] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 3842.563975] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3842.708392] alg: No test for adler32 (adler32-zlib) [ 3843.848839] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 3844.140817] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 3845.799160] Key type lgssc registered [ 3846.938887] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3854.429830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3854.462840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3855.965754] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3855.976178] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3855.993727] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.9/24@tcp [ 3860.164817] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3865.585975] LustreError: 127473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3865.600085] LustreError: 127473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3867.990599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3868.249846] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3868.277526] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.9/24@tcp [ 3868.340906] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3873.707376] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3881.720404] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3886.766443] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3893.800182] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3894.160587] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.369605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 3900.483716] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 3907.345718] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3920.864303] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3920.876677] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3920.886145] LustreError: Skipped 1 previous similar message [ 3920.912276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3923.945195] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3923.948329] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3923.960778] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3926.687555] Lustre: server umount lustre-OST0000 complete [ 3934.178326] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3934.185387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3934.191272] Lustre: Skipped 1 previous similar message [ 3934.209386] Lustre: Skipped 2 previous similar messages [ 3936.222221] Lustre: server umount lustre-MDT0000 complete [ 3939.299279] LustreError: 127467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3939.337140] LustreError: 127467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3940.129740] LustreError: 127452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623152 with bad export cookie 2457630145717332585 [ 3940.136766] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3940.146053] LustreError: 127452:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3940.572262] Lustre: server umount lustre-MDT0001 complete [ 3950.196256] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 3953.274359] Key type lgssc unregistered [ 3953.503083] LNet: 131057:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3953.514563] LNetError: 131057:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3953.528868] LNet: Removed LNI 192.168.203.109@tcp [ 3954.346338] Key type .llcrypt unregistered [ 3954.349254] Key type ._llcrypt unregistered [ 3985.161031] Key type ._llcrypt registered [ 3985.165766] Key type .llcrypt registered [ 3985.305261] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 3999.768901] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4000.852931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4000.906239] alg: No test for adler32 (adler32-zlib) [ 4002.006765] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 4002.221773] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 4003.871223] Key type lgssc registered [ 4005.111479] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4039.531426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4039.571231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4040.878029] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4040.912980] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4040.972131] Lustre: lustre-MDT0000: new disk, initializing [ 4041.032756] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4041.046316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4045.443437] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4056.609460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4056.783821] Lustre: 134454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4056.850986] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4056.854902] Lustre: Skipped 1 previous similar message [ 4056.933285] Lustre: lustre-MDT0001: new disk, initializing [ 4057.054188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4057.112979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4057.127876] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4062.075379] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4071.907617] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4077.893748] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4085.585715] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4085.896364] Lustre: lustre-OST0000: new disk, initializing [ 4085.906938] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4085.917408] Lustre: 136425:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4086.058331] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4091.519718] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4095.585567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4095.600500] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4095.712115] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4101.148926] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4105.533502] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4105.761711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4111.582369] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4111.782316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4116.449730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4116.462083] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4116.465569] LustreError: Skipped 1 previous similar message [ 4116.468125] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4116.480349] Lustre: Skipped 1 previous similar message [ 4120.059443] Lustre: server umount lustre-OST0000 complete [ 4126.695607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4126.703132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4126.725921] Lustre: Skipped 1 previous similar message [ 4126.732898] Lustre: Skipped 1 previous similar message [ 4130.299936] Lustre: server umount lustre-MDT0000 complete [ 4133.720789] LustreError: 134448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623345 with bad export cookie 12962015210794583259 [ 4133.721519] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4133.733316] LustreError: 134448:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4134.079522] Lustre: server umount lustre-MDT0001 complete [ 4142.501872] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4142.901915] LustreError: 138641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.921981] LustreError: 138641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4142.981210] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4147.591892] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4148.195314] LustreError: 138642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4152.292925] LustreError: 138641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4155.763638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4156.120738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4161.300907] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4169.405411] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4175.310952] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4182.936166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4183.275286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4190.199192] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4198.593672] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4203.635933] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4203.865372] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4208.640649] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4208.872598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4216.717943] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4216.935766] Lustre: lustre-OST0001: new disk, initializing [ 4216.938257] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4216.944306] Lustre: 142013:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4217.030286] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4222.970796] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4226.657093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4226.673352] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4226.802979] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4230.533176] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4230.715423] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4235.734057] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4235.980235] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4241.628892] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d1244e55000.ost_server_uuid 50 [ 4243.108259] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d1244e55000.ost_server_uuid in IDLE state after 0 sec [ 4257.769491] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4257.775035] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4257.785829] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4257.785838] Lustre: Skipped 1 previous similar message [ 4257.792825] Lustre: Skipped 1 previous similar message [ 4257.823714] LustreError: Skipped 1 previous similar message [ 4258.961982] Lustre: server umount lustre-OST0001 complete [ 4262.888995] LustreError: 140464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4262.893419] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4262.899700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4262.899712] Lustre: Skipped 1 previous similar message [ 4262.911498] LustreError: 140464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4262.939749] Lustre: Skipped 1 previous similar message [ 4268.001742] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4268.010514] LustreError: 140464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4268.013490] Lustre: Skipped 1 previous similar message [ 4268.031651] LustreError: 140464:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4268.509996] Lustre: server umount lustre-OST0000 complete [ 4274.147937] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4274.156783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4276.198642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4276.213159] Lustre: Skipped 3 previous similar messages [ 4278.282706] Lustre: server umount lustre-MDT0000 complete [ 4281.312868] LustreError: 142596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4281.337309] LustreError: 142596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4281.731850] LustreError: 138623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623493 with bad export cookie 12962015210794584134 [ 4281.740595] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4281.742505] LustreError: 138623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4282.120633] Lustre: server umount lustre-MDT0001 complete [ 4301.330614] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4304.230982] Key type lgssc unregistered [ 4304.680613] LNet: 145013:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4304.695847] LNetError: 145013:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4304.727271] LNet: Removed LNI 192.168.203.109@tcp [ 4305.765137] Key type .llcrypt unregistered [ 4305.783047] Key type ._llcrypt unregistered [ 4351.301289] Key type ._llcrypt registered [ 4351.306803] Key type .llcrypt registered [ 4351.453105] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 4366.264799] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4367.671175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4367.746563] alg: No test for adler32 (adler32-zlib) [ 4368.855901] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 4369.198742] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 4370.888084] Key type lgssc registered [ 4371.840390] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4409.380222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4409.453051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4410.745955] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4410.786568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4410.928914] Lustre: lustre-MDT0000: new disk, initializing [ 4411.015315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4411.032601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4415.506082] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4425.768104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4425.922558] Lustre: 149167:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4426.022856] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4426.031244] Lustre: Skipped 1 previous similar message [ 4426.119515] Lustre: lustre-MDT0001: new disk, initializing [ 4426.221789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4426.255945] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4426.282777] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4430.352936] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4439.185109] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4444.651375] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4451.083558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4451.342288] Lustre: lustre-OST0000: new disk, initializing [ 4451.348626] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4451.353956] Lustre: 151135:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4451.460184] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4456.522862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4456.532853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4456.620541] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4457.104929] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4465.497171] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4470.427786] Lustre: Failing over lustre-OST0000 [ 4470.564832] Lustre: server umount lustre-OST0000 complete [ 4488.365015] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4488.574251] Lustre: lustre-OST0000: Not available for connect from 192.168.203.9@tcp (not set up) [ 4488.632315] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.650628] Lustre: 147390:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788623684/real 1788623684] req@000000002d8dd074 x1875507760102016/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788623700 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4488.713074] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4488.738232] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4489.695375] Lustre: 147388:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788623684/real 1788623684] req@00000000bcd9422c x1875507760102272/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788623700 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4493.658508] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4493.828240] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 4493.829156] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4493.853911] Lustre: 147390:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788623689/real 1788623689] req@0000000062171465 x1875507760103936/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788623705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4493.864591] Lustre: 147390:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4494.567187] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4498.259236] Lustre: Failing over lustre-MDT0000 [ 4498.697427] Lustre: server umount lustre-MDT0000 complete [ 4498.918761] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.940645] Lustre: Skipped 3 previous similar messages [ 4499.940209] Lustre: 147389:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788623694/real 1788623694] req@0000000031693dbc x1875507760104960/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788623710 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4499.996546] Lustre: 147389:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4503.904699] LustreError: 150278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4503.927557] LustreError: 150278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4509.048545] LustreError: 149171:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4509.070396] LustreError: 149171:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4514.143742] LustreError: 149172:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4514.183782] LustreError: 149172:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4518.818945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4518.926664] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4519.263394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4519.310401] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4520.291873] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4523.547908] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4524.534877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4524.543543] Lustre: Skipped 1 previous similar message [ 4524.556055] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4524.607228] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4534.753343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4534.775743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4536.993461] Lustre: server umount lustre-OST0000 complete [ 4544.996207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4544.998581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4545.017173] Lustre: Skipped 2 previous similar messages [ 4545.039355] Lustre: Skipped 2 previous similar messages [ 4547.408428] Lustre: server umount lustre-MDT0000 complete [ 4550.114592] LustreError: 149177:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4550.139791] LustreError: 149177:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4551.233950] LustreError: 149960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788623763 with bad export cookie 267898390167089610 [ 4551.247940] LustreError: 149960:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4551.249294] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4551.659695] Lustre: server umount lustre-MDT0001 complete [ 4563.713931] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 4566.453288] Key type lgssc unregistered [ 4566.703708] LNet: 154789:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4566.717218] LNetError: 154789:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4567.781688] LNet: Removed LNI 192.168.203.109@tcp [ 4568.819789] Key type .llcrypt unregistered [ 4568.822127] Key type ._llcrypt unregistered [ 4604.159879] Key type ._llcrypt registered [ 4604.163672] Key type .llcrypt registered [ 4604.273303] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4605.201615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4605.263650] alg: No test for adler32 (adler32-zlib) [ 4606.346562] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 4606.598504] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 4608.279169] Key type lgssc registered [ 4609.341414] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4615.594670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4615.614383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4616.953860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4620.463860] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4626.403144] LustreError: 156310:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4626.428717] LustreError: 156310:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4628.817788] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4631.524657] LustreError: 156311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.505995] LustreError: 156339:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4633.515431] LustreError: 156339:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4633.519939] LustreError: 156339:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 0, failed: rc = -5 [ 4634.045323] Lustre: server umount lustre-MDT0000 complete [ 4651.078670] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 4661.598649] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 4698.007957] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.218650] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4698.261325] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4698.358977] Lustre: lustre-MDT0000: new disk, initializing [ 4698.454631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4698.476151] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4703.150651] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4713.687977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4713.799545] Lustre: 159643:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4713.862850] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4713.868406] Lustre: Skipped 1 previous similar message [ 4713.960629] Lustre: lustre-MDT0001: new disk, initializing [ 4714.074863] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4714.109856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4714.127521] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4718.713533] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4727.972443] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4733.523631] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4739.725785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4739.931897] Lustre: lustre-OST0000: new disk, initializing [ 4739.935465] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4739.940907] Lustre: 161613:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4740.008369] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4745.865583] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 4750.415131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4750.426626] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4750.505670] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4754.927108] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4759.644157] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4761.638232] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:59:31 (1788623971) [ 5021.151948] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5021.164533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.172402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5022.176691] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5022.180761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5022.208111] Lustre: Skipped 2 previous similar messages [ 5027.310729] LustreError: 159657:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5027.324793] LustreError: 159657:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5027.404709] Lustre: server umount lustre-MDT0000 complete [ 5030.716039] LustreError: 159638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624242 with bad export cookie 17705197951402018983 [ 5030.719807] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5030.729595] LustreError: 159638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5032.417158] LustreError: 160454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5032.418253] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5032.460838] Lustre: Skipped 1 previous similar message [ 5032.467217] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5035.436458] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5040.543902] LustreError: 159650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5040.544400] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5040.578497] LustreError: 159650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5045.218781] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5045.440920] Lustre: server umount lustre-MDT0001 complete [ 5055.496300] Lustre: server umount lustre-OST0000 complete [ 5064.734300] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 5077.055775] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 5114.858897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5115.072377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5115.118391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5115.198299] Lustre: lustre-MDT0000: new disk, initializing [ 5115.297850] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5115.326764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5119.537076] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5130.155182] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5130.219271] Lustre: 166290:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5130.258978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5130.261783] Lustre: Skipped 1 previous similar message [ 5130.366432] Lustre: lustre-MDT0001: new disk, initializing [ 5130.464434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5130.507500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5130.518149] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5134.587317] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5144.266983] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5152.158914] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5161.241897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5161.565638] Lustre: lustre-OST0000: new disk, initializing [ 5161.572266] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5161.590488] Lustre: 168262:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5161.774476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5168.790360] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5168.803718] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5168.939647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5170.412662] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5180.164982] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5185.309929] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5185.476575] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5190.793958] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5191.061045] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5194.727647] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5194.733289] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5194.753257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5194.760411] Lustre: Skipped 1 previous similar message [ 5199.083407] Lustre: server umount lustre-OST0000 complete [ 5204.960277] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.971590] Lustre: Skipped 1 previous similar message [ 5214.247578] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5214.258401] Lustre: Skipped 6 previous similar messages [ 5217.247169] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5217.549438] Lustre: server umount lustre-MDT0000 complete [ 5219.304072] LustreError: 166304:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5219.319194] LustreError: 166304:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5221.243733] LustreError: 166284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624433 with bad export cookie 17705197951402177813 [ 5221.245726] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5221.252608] LustreError: 166284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5221.765423] Lustre: server umount lustre-MDT0001 complete [ 5229.758021] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:07:20 (1788624440) [ 5245.929409] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 5255.567778] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 5290.051344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5290.314675] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5290.336597] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5290.412589] Lustre: lustre-MDT0000: new disk, initializing [ 5290.481529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5290.501125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5294.362743] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5303.646686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5303.853143] Lustre: 173019:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5304.187673] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5308.148298] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5317.312742] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5323.418390] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5330.228767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5330.597927] Lustre: lustre-OST0000: new disk, initializing [ 5330.602910] Lustre: Skipped 1 previous similar message [ 5330.605321] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5330.619691] Lustre: Skipped 2 previous similar messages [ 5330.627033] Lustre: 174990:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5337.095829] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5337.664812] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5337.675865] Lustre: Skipped 1 previous similar message [ 5337.683246] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5337.775593] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5346.462031] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5358.568710] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5358.572801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5358.579354] LustreError: Skipped 2 previous similar messages [ 5358.583399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5358.601911] Lustre: Skipped 1 previous similar message [ 5368.291577] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5368.506882] Lustre: server umount lustre-OST0000 complete [ 5387.231147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5387.483067] Lustre: server umount lustre-MDT0000 complete [ 5388.262534] LustreError: 175871:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5388.284139] LustreError: 175871:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5391.274115] LustreError: 175809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624603 with bad export cookie 17705197951402178695 [ 5391.286276] LustreError: 175809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5391.289780] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5391.655339] Lustre: server umount lustre-MDT0001 complete [ 5400.806546] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:10:11 (1788624611) [ 5420.422594] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 5430.746412] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 5466.719905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5466.968576] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5467.009569] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5467.155690] Lustre: lustre-MDT0000: new disk, initializing [ 5467.229399] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5467.240530] Lustre: Skipped 2 previous similar messages [ 5467.253408] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5472.104240] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5483.356249] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5483.473815] Lustre: 179317:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5483.758125] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5488.979255] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5498.599495] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5504.928605] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5512.708940] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5513.104326] Lustre: 181295:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5520.179193] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5521.981309] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5522.136899] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5530.824194] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5542.881791] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5542.897805] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5542.912258] Lustre: Skipped 2 previous similar messages [ 5542.916356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5542.920520] Lustre: Skipped 10 previous similar messages [ 5545.379672] Lustre: server umount lustre-OST0000 complete [ 5550.573279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5550.578352] LustreError: Skipped 1 previous similar message [ 5555.596116] Lustre: server umount lustre-MDT0000 complete [ 5558.244254] LustreError: 179326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5558.265631] LustreError: 179326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5559.591763] LustreError: 179311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624771 with bad export cookie 17705197951402179780 [ 5559.598365] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5559.603760] LustreError: 179311:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5560.008638] Lustre: server umount lustre-MDT0001 complete [ 5571.978818] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 5575.083288] Key type lgssc unregistered [ 5575.455444] LNet: 183152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5575.465868] LNetError: 183152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5575.491994] LNet: Removed LNI 192.168.203.109@tcp [ 5576.554184] Key type .llcrypt unregistered [ 5576.566209] Key type ._llcrypt unregistered [ 5600.441776] Key type ._llcrypt registered [ 5600.447509] Key type .llcrypt registered [ 5600.533696] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 5601.431093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5601.537077] alg: No test for adler32 (adler32-zlib) [ 5602.642465] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 5602.897029] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 5604.584103] Key type lgssc registered [ 5605.816426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5613.641795] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5613.670946] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5615.172073] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5619.420430] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5625.312469] LustreError: 184566:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5625.333458] LustreError: 184566:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5627.813880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5628.367418] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5632.652606] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5641.971690] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5650.366470] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5657.342591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5657.834532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5660.934676] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5664.856099] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5670.968278] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5693.922977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5693.928811] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5693.935183] Lustre: Skipped 1 previous similar message [ 5693.936196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5693.944156] Lustre: Skipped 1 previous similar message [ 5699.043555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5699.331509] Lustre: server umount lustre-OST0000 complete [ 5704.168022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5704.170556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5704.183498] Lustre: Skipped 1 previous similar message [ 5704.190300] Lustre: Skipped 1 previous similar message [ 5709.283437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5709.294115] Lustre: Skipped 2 previous similar messages [ 5709.863568] Lustre: server umount lustre-MDT0000 complete [ 5713.951822] LustreError: 184548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788624925 with bad export cookie 10206837103748994697 [ 5713.967128] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5713.977090] LustreError: 184548:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5714.917817] Lustre: server umount lustre-MDT0001 complete [ 5730.123558] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 5733.119201] Key type lgssc unregistered [ 5733.471067] LNet: 188122:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5733.476994] LNetError: 188122:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5733.508263] LNet: Removed LNI 192.168.203.109@tcp [ 5734.499751] Key type .llcrypt unregistered [ 5734.501910] Key type ._llcrypt unregistered [ 5759.518196] Key type ._llcrypt registered [ 5759.528990] Key type .llcrypt registered [ 5759.661252] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 5760.854028] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5761.082292] alg: No test for adler32 (adler32-zlib) [ 5762.283944] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 5762.698460] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 5764.407740] Key type lgssc registered [ 5765.383143] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5772.488733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5772.517674] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5773.896616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5778.650420] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5783.013165] LustreError: 189536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5783.035774] LustreError: 189536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5787.236453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5787.685994] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5791.928752] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5798.987420] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5804.484919] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5811.668633] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5812.050307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5818.234285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5818.559270] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5826.345451] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5833.702982] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5833.703206] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5833.707971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5833.734475] Lustre: Skipped 1 previous similar message [ 5838.897766] Lustre: server umount lustre-OST0000 complete [ 5845.933215] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8d1250d6c800.ost_server_uuid 50 [ 5847.448315] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5863.397561] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5863.400566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5863.418095] Lustre: Skipped 1 previous similar message [ 5863.439110] Lustre: Skipped 3 previous similar messages [ 5867.580248] Lustre: server umount lustre-MDT0000 complete [ 5868.515175] LustreError: 189536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5868.539398] LustreError: 189536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5871.032561] LustreError: 189517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625083 with bad export cookie 3070963234263265511 [ 5871.033708] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5871.044066] LustreError: 189517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5871.317379] Lustre: server umount lustre-MDT0001 complete [ 5880.147885] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:18:10 (1788625090) [ 5887.046814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5887.662254] LustreError: 193449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5887.753856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5892.222143] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5893.088163] LustreError: 193450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5897.189990] LustreError: 193449:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5900.107233] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5900.477175] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5904.727981] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5911.957997] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5917.803499] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5925.107265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5925.369662] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5931.629436] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5932.602659] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5938.708975] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5944.696169] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5944.946371] Lustre: lustre-OST0001: new disk, initializing [ 5944.948841] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5944.955106] Lustre: 196291:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5945.109986] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5952.201326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5952.226360] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5952.376431] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5952.637241] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 5963.652332] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5967.342086] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5967.357235] LustreError: Skipped 1 previous similar message [ 5967.362407] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5967.382572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5972.472581] Lustre: server umount lustre-OST0000 complete [ 5972.970879] LustreError: 195270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5972.996795] LustreError: 195270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5978.841209] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5979.015755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5983.824986] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5983.951058] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5989.743162] LustreError: 195272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5989.778159] LustreError: 195272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6002.775369] Lustre: server umount lustre-OST0001 complete [ 6008.288537] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6008.302757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6008.326458] Lustre: Skipped 1 previous similar message [ 6008.337759] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6008.347963] Lustre: Skipped 2 previous similar messages [ 6013.920205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6013.927890] Lustre: Skipped 2 previous similar messages [ 6018.104435] Lustre: server umount lustre-MDT0000 complete [ 6019.871663] Lustre: 188848:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788625215/real 1788625215] req@0000000022ebb68d x1875509220829952/t0(0) o13->lustre-OST0001-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788625231 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 6019.931561] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6019.949993] Lustre: Skipped 1 previous similar message [ 6021.678598] LustreError: 193431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625233 with bad export cookie 3070963234263266582 [ 6021.681229] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6021.698420] LustreError: 193431:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6022.027143] Lustre: server umount lustre-MDT0001 complete [ 6048.732271] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:20:58 (1788625258) [ 6056.817817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6057.180374] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6057.212638] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6057.416931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6061.847275] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6069.679966] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6069.746762] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6069.776122] Lustre: 199733:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6074.378486] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6081.882634] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6089.179272] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6096.279243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6096.544134] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6103.113245] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6103.563728] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6111.346392] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6118.926380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6118.990217] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6125.291857] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6133.176775] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6136.802326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.807797] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6136.835871] Lustre: Skipped 1 previous similar message [ 6136.846859] Lustre: Skipped 1 previous similar message [ 6142.029612] Lustre: server umount lustre-OST0000 complete [ 6144.347365] LustreError: 201586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6144.393022] LustreError: 201586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6157.284715] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6157.287742] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.291651] LustreError: Skipped 1 previous similar message [ 6157.292517] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6157.292521] Lustre: Skipped 3 previous similar messages [ 6157.311043] Lustre: Skipped 1 previous similar message [ 6163.609818] Lustre: server umount lustre-OST0001 complete [ 6167.520626] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6173.552545] Lustre: server umount lustre-MDT0000 complete [ 6177.058895] LustreError: 200481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625389 with bad export cookie 3070963234263267989 [ 6177.070678] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6177.070177] LustreError: 200481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6177.423375] Lustre: server umount lustre-MDT0001 complete [ 6200.409107] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:23:30 (1788625410) [ 6217.099824] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 6228.179304] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 6263.616334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6263.813890] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6263.883172] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6263.952724] Lustre: lustre-MDT0000: new disk, initializing [ 6264.034217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6264.040120] Lustre: Skipped 3 previous similar messages [ 6264.052464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6268.329692] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6278.388025] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6278.516718] Lustre: 207832:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6278.557722] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6278.561761] Lustre: Skipped 1 previous similar message [ 6278.625257] Lustre: lustre-MDT0001: new disk, initializing [ 6278.640459] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6278.644520] Lustre: Skipped 7 previous similar messages [ 6278.768076] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6278.774857] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6283.311693] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6291.799838] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6297.502562] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6304.669816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6305.093393] Lustre: lustre-OST0000: new disk, initializing [ 6305.098885] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6305.108643] Lustre: 209801:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6311.768247] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6313.628115] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6313.635233] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6313.750547] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6322.110482] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6326.908506] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6327.113390] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6332.359311] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6332.591978] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6339.561177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6339.562940] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6339.582959] Lustre: Skipped 3 previous similar messages [ 6340.734372] Lustre: server umount lustre-OST0000 complete [ 6345.190424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6345.195748] Lustre: Skipped 2 previous similar messages [ 6351.236270] Lustre: server umount lustre-MDT0000 complete [ 6354.918648] LustreError: 207841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6354.940810] LustreError: 207841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 6355.389761] LustreError: 209800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625567 with bad export cookie 3070963234263269543 [ 6355.395711] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6355.402081] LustreError: 209800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6355.598496] Lustre: server umount lustre-MDT0001 complete [ 6364.224926] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6369.668307] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6378.598342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6384.346449] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6392.410192] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6398.527068] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6406.052957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.393053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6406.401089] Lustre: Skipped 4 previous similar messages [ 6413.004764] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6421.316719] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6426.302558] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6426.510513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6431.353564] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6431.591022] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6434.275024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6434.277095] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6434.292276] Lustre: Skipped 2 previous similar messages [ 6434.312652] LustreError: Skipped 2 previous similar messages [ 6439.609897] Lustre: server umount lustre-OST0000 complete [ 6446.833155] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6451.834980] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6481.173241] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6488.509283] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6489.775896] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.811563] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6489.818301] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6489.851177] Lustre: Skipped 1 previous similar message [ 6496.755295] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6499.644529] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 1 sec [ 6505.445126] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6505.450556] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6505.453492] LustreError: Skipped 1 previous similar message [ 6505.469754] Lustre: Skipped 7 previous similar messages [ 6517.217265] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6517.382662] Lustre: server umount lustre-OST0000 complete [ 6528.962598] Lustre: server umount lustre-MDT0000 complete [ 6531.048822] LustreError: 212014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6531.073221] LustreError: 212014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6533.964562] LustreError: 212001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625745 with bad export cookie 3070963234263270432 [ 6533.990301] LustreError: 212001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6534.001638] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6534.708131] Lustre: server umount lustre-MDT0001 complete [ 6545.954776] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:29:16 (1788625756) [ 6552.706226] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6557.504354] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6565.660463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.755716] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6577.907647] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6583.480295] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6589.982513] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6596.360968] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6604.606317] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6609.443849] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6609.668036] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6614.442664] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6614.665608] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6621.230369] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6621.363665] Lustre: lustre-OST0001: new disk, initializing [ 6621.367395] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6621.371820] Lustre: 221360:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6627.126726] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6628.466321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6628.482299] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6628.581191] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6636.275696] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6640.715412] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6640.924620] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6645.647196] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6645.858628] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6649.327845] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6649.329772] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6649.333113] LustreError: Skipped 1 previous similar message [ 6649.353935] Lustre: Skipped 6 previous similar messages [ 6662.112027] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6662.498094] Lustre: server umount lustre-OST0001 complete [ 6670.778444] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6670.962470] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6676.615833] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6676.916408] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6706.896852] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6707.228942] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6707.242103] Lustre: Skipped 5 previous similar messages [ 6712.298381] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6712.312961] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6712.317789] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6713.300437] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6720.278740] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6722.230825] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6730.438716] Lustre: server umount lustre-OST0001 complete [ 6755.115238] Lustre: server umount lustre-OST0000 complete [ 6763.497152] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6763.505749] Lustre: Skipped 21 previous similar messages [ 6764.967196] Lustre: server umount lustre-MDT0000 complete [ 6768.654209] LustreError: 219831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788625980 with bad export cookie 3070963234263271398 [ 6768.658705] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6768.666865] LustreError: 219831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6775.150387] Lustre: server umount lustre-MDT0001 complete [ 6799.750639] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:33:29 (1788626009) [ 6807.356985] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6807.759389] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6807.810412] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6812.772984] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6822.095354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6822.238152] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6822.278163] Lustre: 226608:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6827.947687] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6837.181500] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6842.895314] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6851.105183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6851.220951] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6862.235521] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6873.189131] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6883.178157] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6883.233291] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6890.710286] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 6899.126758] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6901.159909] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6906.968106] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6907.257464] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6912.871379] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6913.139933] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6921.559645] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8d1259a76800.ost_server_uuid 50 [ 6923.670744] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8d1259a76800.ost_server_uuid in IDLE state after 0 sec [ 6925.551772] Lustre: Permanently deactivating lustre-OST0001 [ 6925.613528] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6930.066700] Lustre: Permanently reactivating lustre-OST0001 [ 6930.098260] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 6930.108840] Lustre: Skipped 2 previous similar messages [ 6936.041432] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 6936.054655] LustreError: Skipped 6 previous similar messages [ 6936.063435] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6936.082209] Lustre: Skipped 6 previous similar messages [ 6940.779381] Lustre: server umount lustre-OST0001 complete [ 6942.694237] LustreError: 228456:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6942.711403] LustreError: 228456:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 40 previous similar messages [ 6951.005451] Lustre: server umount lustre-OST0000 complete [ 6961.735324] Lustre: server umount lustre-MDT0000 complete [ 6965.741853] LustreError: 226598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626177 with bad export cookie 3070963234263272686 [ 6965.756417] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6965.769588] LustreError: 226598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6966.354340] Lustre: server umount lustre-MDT0001 complete [ 6991.676321] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:36:41 (1788626201) [ 6999.753071] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6999.958119] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6999.980431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6999.986314] Lustre: Skipped 2 previous similar messages [ 7005.278714] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7015.017303] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7015.148809] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7015.194168] Lustre: 233304:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7020.075706] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7027.643624] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7033.347162] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7040.239991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7040.549767] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7040.590616] Lustre: Permanently deactivating lustre-OST0000 [ 7040.617468] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7046.212781] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7047.455311] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7055.052243] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7061.887271] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7062.001766] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7068.965371] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7072.456130] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7077.544734] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7079.675228] Lustre: Permanently reactivating lustre-OST0000 [ 7079.707870] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7079.713765] Lustre: Skipped 2 previous similar messages [ 7088.987764] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 00000000ab3eac78, cur 1788626300 deadline 1788626288 last 1788626258 [ 7089.231196] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7089.238642] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7089.250600] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7089.259850] Lustre: Skipped 1 previous similar message [ 7089.325531] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7100.258132] Lustre: server umount lustre-OST0001 complete [ 7110.783317] Lustre: server umount lustre-OST0000 complete [ 7121.267765] Lustre: server umount lustre-MDT0000 complete [ 7125.993717] LustreError: 234046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626337 with bad export cookie 3070963234263274352 [ 7125.995407] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7126.006321] LustreError: 234046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7126.495646] Lustre: server umount lustre-MDT0001 complete [ 7140.225981] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 7143.510666] Key type lgssc unregistered [ 7143.862718] LNet: 238184:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7143.873705] LNetError: 238184:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7143.899204] LNet: Removed LNI 192.168.203.109@tcp [ 7144.902570] Key type .llcrypt unregistered [ 7144.908328] Key type ._llcrypt unregistered [ 7171.754343] Key type ._llcrypt registered [ 7171.765834] Key type .llcrypt registered [ 7171.956628] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7173.194303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7173.513436] alg: No test for adler32 (adler32-zlib) [ 7174.776329] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 7175.108565] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 7176.855211] Key type lgssc registered [ 7178.272920] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7188.706095] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7188.750583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7190.312049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7196.093485] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7200.737142] LustreError: 239650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7200.771857] LustreError: 239650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7205.862969] LustreError: 239649:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7206.288309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7206.493071] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7206.538158] Lustre: 239638:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7206.557529] Lustre: Permanently deactivating lustre-MDT0001 [ 7206.586612] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7206.974039] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7206.990551] LustreError: 239680:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7207.011971] LustreError: 239680:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7207.030913] LustreError: 239680:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 17, retries 1, failed: rc = -108 [ 7207.072094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7212.151243] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7220.794135] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7228.713235] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7236.631371] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7237.088317] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7244.911434] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7251.428909] LustreError: 241489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7254.275126] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7256.556512] LustreError: 242107:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7260.667230] LustreError: 241489:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7260.690783] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7260.712110] LustreError: 241489:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7262.426547] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7262.612741] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7267.854552] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7267.861341] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7270.611588] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7279.276834] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7283.560165] Lustre: Permanently reactivating lustre-MDT0001 [ 7283.590947] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7283.594056] Lustre: Skipped 2 previous similar messages [ 7285.132660] Lustre: lustre-MDT0001: Client 9ac1ac57-4e95-452d-b05e-14c8547b2084 (at 192.168.203.9@tcp) reconnecting [ 7292.398421] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7294.293892] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7300.548856] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7300.798783] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7305.635320] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7305.873560] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7308.694637] Lustre: Permanently deactivating lustre-MDT0001 [ 7308.718082] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7308.721862] Lustre: Skipped 2 previous similar messages [ 7315.556743] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7324.132746] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7324.155515] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7324.169908] Lustre: Skipped 1 previous similar message [ 7324.191757] Lustre: Skipped 3 previous similar messages [ 7326.538630] Lustre: server umount lustre-MDT0000 complete [ 7329.251172] LustreError: 239650:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7329.269660] LustreError: 239650:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7330.937036] LustreError: 242504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626542 with bad export cookie 17795634375668031630 [ 7330.941799] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7330.957248] LustreError: 242504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7331.526500] Lustre: server umount lustre-MDT0001 complete [ 7337.901577] Lustre: server umount lustre-OST0000 complete [ 7351.298699] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:42:41 (1788626561) [ 7364.022995] Lustre: server umount lustre-OST0001 complete [ 7371.160911] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 7381.824237] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7424.414576] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7424.759525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7424.779025] Lustre: Skipped 2 previous similar messages [ 7424.807470] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7424.890742] Lustre: lustre-MDT0000: new disk, initializing [ 7425.023172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7425.048255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7430.708119] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7443.251104] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7443.439944] Lustre: 247907:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7443.519669] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7443.523872] Lustre: Skipped 1 previous similar message [ 7443.629686] Lustre: lustre-MDT0001: new disk, initializing [ 7443.797498] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7443.837069] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7443.855578] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7449.674405] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7460.983684] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7467.768530] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7475.091296] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7475.313749] Lustre: lustre-OST0000: new disk, initializing [ 7475.316121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7475.319194] Lustre: 249880:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7475.399201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7481.918754] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7481.940445] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7481.942570] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7482.070308] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7491.390333] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7493.926462] LustreError: 247917:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7503.782944] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7503.959093] LustreError: 247917:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7504.008417] LustreError: 247915:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7504.106214] Lustre: lustre-OST0001: new disk, initializing [ 7504.115566] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7504.122059] Lustre: 250993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7510.654862] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7510.677748] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7510.844828] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7511.545945] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7514.111150] LustreError: 247915:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7521.332069] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7522.895965] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7526.386452] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7526.392908] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7526.395513] LustreError: Skipped 1 previous similar message [ 7526.397111] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7526.406826] Lustre: Skipped 3 previous similar messages [ 7526.428879] Lustre: Skipped 1 previous similar message [ 7530.668772] Lustre: server umount lustre-OST0001 complete [ 7531.508305] LustreError: 249862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7531.527800] LustreError: 249862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7548.903954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7548.905280] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7548.914031] LustreError: Skipped 1 previous similar message [ 7548.916460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7548.931161] Lustre: Skipped 1 previous similar message [ 7548.941223] Lustre: Skipped 1 previous similar message [ 7551.971291] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7551.981220] Lustre: Skipped 1 previous similar message [ 7553.367466] Lustre: server umount lustre-OST0000 complete [ 7561.698864] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7561.718162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7561.734106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7564.648980] Lustre: server umount lustre-MDT0000 complete [ 7567.334673] LustreError: 251002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7567.361422] LustreError: 251002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 7569.283253] LustreError: 247902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788626781 with bad export cookie 17795634375668034227 [ 7569.284287] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7569.298871] LustreError: 247902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7575.962637] Lustre: server umount lustre-MDT0001 complete [ 7593.178340] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 7596.809885] Key type lgssc unregistered [ 7597.170709] LNet: 253046:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7597.180970] LNetError: 253046:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7598.256278] LNet: Removed LNI 192.168.203.109@tcp [ 7598.953464] Key type .llcrypt unregistered [ 7598.955637] Key type ._llcrypt unregistered [ 7640.335693] Key type ._llcrypt registered [ 7640.342748] Key type .llcrypt registered [ 7640.475873] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7641.587373] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7641.630934] alg: No test for adler32 (adler32-zlib) [ 7642.821366] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 7643.157166] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 7644.863288] Key type lgssc registered [ 7645.923408] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7653.525208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7653.540592] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7654.789760] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7654.839499] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7655.026810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7659.728056] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7668.120459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7668.217326] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7668.248168] Lustre: 255249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7668.573773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7674.540932] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7684.545570] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7692.042980] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7699.222490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7699.337246] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7699.488092] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7705.849617] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7706.523912] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7716.195627] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7772.657123] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7772.676617] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7772.692744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7777.251500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7777.261966] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7777.277582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7778.576246] Lustre: server umount lustre-OST0000 complete [ 7783.249809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7798.501455] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7798.866801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7802.978195] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7802.999337] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7806.014900] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7815.122296] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7817.324527] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7817.350746] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7858.666851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7858.674551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7858.707615] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7858.707624] Lustre: Skipped 1 previous similar message [ 7862.493182] Lustre: server umount lustre-OST0000 complete [ 7868.384129] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7868.393519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7868.406175] Lustre: Skipped 1 previous similar message [ 7868.417482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7868.419798] Lustre: Skipped 1 previous similar message [ 7873.435800] Lustre: server umount lustre-MDT0000 complete [ 7877.921078] LustreError: 255241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627089 with bad export cookie 14219645855673244968 [ 7877.921759] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7877.937385] LustreError: 255241:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7878.534679] Lustre: server umount lustre-MDT0001 complete [ 7891.529890] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 7894.505981] Key type lgssc unregistered [ 7894.841802] LNet: 260670:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7894.847763] LNetError: 260670:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7894.863642] LNet: Removed LNI 192.168.203.109@tcp [ 7895.944585] Key type .llcrypt unregistered [ 7895.950142] Key type ._llcrypt unregistered [ 7924.869809] Key type ._llcrypt registered [ 7924.872355] Key type .llcrypt registered [ 7924.990340] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 7926.469831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7926.510503] alg: No test for adler32 (adler32-zlib) [ 7927.782304] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 7928.100844] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 7929.871286] Key type lgssc registered [ 7931.264753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7941.021514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7941.034089] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7942.584899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7947.518880] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7952.870217] LustreError: 262086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7952.893804] LustreError: 262086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7957.221246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7957.601480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7962.917788] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 7971.881793] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7978.191160] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7986.062072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7986.745216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7987.769432] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7994.590640] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8004.162542] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8032.736693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8032.740534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8032.750063] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8032.779085] Lustre: Skipped 1 previous similar message [ 8037.060908] Lustre: server umount lustre-OST0000 complete [ 8042.979702] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8043.007405] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8043.015284] Lustre: Skipped 2 previous similar messages [ 8048.099222] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8048.105670] Lustre: Skipped 1 previous similar message [ 8052.196134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8052.208814] Lustre: Skipped 1 previous similar message [ 8055.263648] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8055.978886] Lustre: server umount lustre-MDT0000 complete [ 8057.313883] LustreError: 262080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8057.325565] LustreError: 262080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8060.421959] LustreError: 262066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627272 with bad export cookie 10367650360290156443 [ 8060.439489] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8060.448103] LustreError: 262066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8061.011710] Lustre: server umount lustre-MDT0001 complete [ 8075.484072] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 8078.646665] Key type lgssc unregistered [ 8078.896591] LNet: 266392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8078.901580] LNetError: 266392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8078.915663] LNet: Removed LNI 192.168.203.109@tcp [ 8079.972702] Key type .llcrypt unregistered [ 8079.976294] Key type ._llcrypt unregistered [ 8101.423413] Key type ._llcrypt registered [ 8101.426539] Key type .llcrypt registered [ 8101.563052] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8102.641336] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8102.723235] alg: No test for adler32 (adler32-zlib) [ 8103.800376] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 8104.153500] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 8105.872864] Key type lgssc registered [ 8106.965778] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8115.214311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8115.261967] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8116.768842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8121.310987] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8126.947833] LustreError: 267609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8126.976550] LustreError: 267609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8131.163644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8131.501318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8135.845174] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8145.853531] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8153.149211] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8160.547129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8160.978763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8162.045801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8168.119237] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8176.975757] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8191.974201] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8191.994181] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8191.999462] Lustre: Skipped 1 previous similar message [ 8196.503285] Lustre: server umount lustre-OST0000 complete [ 8202.208558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8202.223821] Lustre: Skipped 1 previous similar message [ 8202.228187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8207.087127] Lustre: server umount lustre-MDT0000 complete [ 8207.334047] LustreError: 267605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8207.356172] LustreError: 267605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8211.141082] LustreError: 267591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627423 with bad export cookie 1660186408757593073 [ 8211.146095] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8211.161047] LustreError: 267591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8211.451665] Lustre: server umount lustre-MDT0001 complete [ 8223.500876] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 8226.560921] Key type lgssc unregistered [ 8226.883684] LNet: 271286:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8226.891980] LNetError: 271286:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8226.919211] LNet: Removed LNI 192.168.203.109@tcp [ 8228.032522] Key type .llcrypt unregistered [ 8228.034640] Key type ._llcrypt unregistered [ 8254.306128] Key type ._llcrypt registered [ 8254.307867] Key type .llcrypt registered [ 8254.440716] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8255.398565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8255.570034] alg: No test for adler32 (adler32-zlib) [ 8256.721539] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 8257.117050] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 8258.841800] Key type lgssc registered [ 8260.083373] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8268.414710] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8268.449371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8269.932720] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8274.949652] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8279.528351] LustreError: 272702:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8279.563066] LustreError: 272702:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8284.639539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.643312] LustreError: 272703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8285.040351] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8290.268625] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8298.117870] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8305.624441] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8313.819816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8314.315390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8315.404396] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8322.140247] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8330.511333] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8360.416488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8360.423200] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8360.434083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8360.451913] Lustre: Skipped 1 previous similar message [ 8362.722297] Lustre: server umount lustre-OST0000 complete [ 8367.075119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8367.099824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8367.125973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8367.135021] Lustre: Skipped 1 previous similar message [ 8370.656243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8370.660546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8370.690557] Lustre: Skipped 1 previous similar message [ 8372.885224] Lustre: server umount lustre-MDT0000 complete [ 8375.778802] LustreError: 272703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8375.808333] LustreError: 272703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8377.327876] LustreError: 272682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627589 with bad export cookie 1398367590625282086 [ 8377.337218] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8377.355721] LustreError: 272682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8377.753688] Lustre: server umount lustre-MDT0001 complete [ 8390.561788] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 8393.610297] Key type lgssc unregistered [ 8393.934135] LNet: 277010:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8393.943043] LNetError: 277010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8393.972845] LNet: Removed LNI 192.168.203.109@tcp [ 8394.832879] Key type .llcrypt unregistered [ 8394.835524] Key type ._llcrypt unregistered [ 8413.721552] Key type ._llcrypt registered [ 8413.726764] Key type .llcrypt registered [ 8413.812848] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8414.881250] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8414.994603] alg: No test for adler32 (adler32-zlib) [ 8416.062611] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 8416.277388] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 8418.008402] Key type lgssc registered [ 8419.253238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8427.157069] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8427.177756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8428.580598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8432.556119] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8438.755779] LustreError: 278244:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8438.768510] LustreError: 278244:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8440.969470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8441.350284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8446.296613] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8454.388180] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8461.401430] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8468.307061] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8468.631173] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8473.789569] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8476.163972] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8484.299317] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8499.680826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8499.696160] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8499.717370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8501.737976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8501.747274] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8501.767992] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8504.988291] Lustre: server umount lustre-OST0000 complete [ 8511.970703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8511.978625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8511.988972] Lustre: Skipped 1 previous similar message [ 8512.012984] Lustre: Skipped 2 previous similar messages [ 8517.092278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8517.110741] Lustre: Skipped 1 previous similar message [ 8521.186125] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8523.232685] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8523.404197] Lustre: server umount lustre-MDT0000 complete [ 8526.315708] LustreError: 278233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8526.357109] LustreError: 278233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8527.041355] LustreError: 278209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627739 with bad export cookie 7101909384302519426 [ 8527.057655] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8527.310801] Lustre: server umount lustre-MDT0001 complete [ 8539.329557] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 8542.364751] Key type lgssc unregistered [ 8542.672554] LNet: 281957:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8542.684514] LNetError: 281957:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8542.701437] LNet: Removed LNI 192.168.203.109@tcp [ 8543.844635] Key type .llcrypt unregistered [ 8543.847904] Key type ._llcrypt unregistered [ 8555.768844] Key type ._llcrypt registered [ 8555.772029] Key type .llcrypt registered [ 8555.910131] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8573.670332] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 8591.393809] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8592.988846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8593.443533] alg: No test for adler32 (adler32-zlib) [ 8594.726517] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 8595.126124] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 8596.905878] Key type lgssc registered [ 8598.169521] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8634.975327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8635.004249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8636.302223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8636.331543] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8636.424972] Lustre: lustre-MDT0000: new disk, initializing [ 8636.502023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8636.520062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8642.530547] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8654.494913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8654.677750] Lustre: 285512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8654.788830] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8654.797443] Lustre: Skipped 1 previous similar message [ 8654.990372] Lustre: lustre-MDT0001: new disk, initializing [ 8655.156488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8655.237257] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8655.256082] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8660.051469] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8670.238731] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8676.997164] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8684.864757] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8685.176322] Lustre: lustre-OST0000: new disk, initializing [ 8685.178590] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8685.183119] Lustre: 287479:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8685.252677] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8691.965384] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8694.867227] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8694.891046] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8694.988546] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8702.027613] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8707.307797] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8707.530699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8712.305806] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8712.498825] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8715.748381] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8715.757732] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8715.758817] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8715.772108] Lustre: Skipped 1 previous similar message [ 8720.714828] Lustre: server umount lustre-OST0000 complete [ 8725.985942] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8725.989456] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8726.000826] Lustre: Skipped 1 previous similar message [ 8726.013356] Lustre: Skipped 2 previous similar messages [ 8731.105068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8731.113302] Lustre: Skipped 1 previous similar message [ 8735.203385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8735.209238] Lustre: Skipped 1 previous similar message [ 8739.295123] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8739.577556] Lustre: server umount lustre-MDT0000 complete [ 8743.607486] LustreError: 287490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788627955 with bad export cookie 7598068593003872501 [ 8743.614274] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8743.619447] LustreError: 287490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8743.973580] Lustre: server umount lustre-MDT0001 complete [ 8753.687210] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:06:03 (1788627963) [ 8760.814268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8761.299034] LustreError: 289895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8761.318788] LustreError: 289895:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8761.396686] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8766.450973] LustreError: 289896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8766.787941] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8771.565634] LustreError: 289895:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8775.109773] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8775.495461] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8780.299894] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8787.786760] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8793.459270] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8799.647050] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8800.006838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8805.828210] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8813.409562] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8826.857373] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8826.867214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8826.868048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8826.880677] LustreError: Skipped 1 previous similar message [ 8829.563746] Lustre: server umount lustre-OST0000 complete [ 8837.092126] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8837.093957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8837.115386] Lustre: Skipped 1 previous similar message [ 8837.116933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8837.138332] Lustre: Skipped 2 previous similar messages [ 8840.003126] Lustre: server umount lustre-MDT0000 complete [ 8842.211032] LustreError: 289891:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8842.233470] LustreError: 289891:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8844.352377] LustreError: 289875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628056 with bad export cookie 7598068593003873383 [ 8844.354342] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8844.367235] LustreError: 289875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8844.853457] Lustre: server umount lustre-MDT0001 complete [ 8859.059628] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 8863.315742] Key type lgssc unregistered [ 8863.759601] LNet: 293430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8863.772737] LNetError: 293430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8863.787893] LNet: Removed LNI 192.168.203.109@tcp [ 8864.936341] Key type .llcrypt unregistered [ 8864.940940] Key type ._llcrypt unregistered [ 8893.974708] Key type ._llcrypt registered [ 8893.978026] Key type .llcrypt registered [ 8894.133845] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 8909.478555] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 8910.265929] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8910.320494] alg: No test for adler32 (adler32-zlib) [ 8911.398900] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 8911.625376] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 8913.327589] Key type lgssc registered [ 8914.375022] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8960.433957] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8960.481837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8961.796712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8961.851954] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8961.992337] Lustre: lustre-MDT0000: new disk, initializing [ 8962.105983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8962.160617] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8967.992654] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8979.591686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8979.762589] Lustre: 297252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8979.838040] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8979.842284] Lustre: Skipped 1 previous similar message [ 8979.908754] Lustre: lustre-MDT0001: new disk, initializing [ 8980.004385] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8980.052719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8980.071829] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8983.934649] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 8994.022680] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9001.006932] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9008.654364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9009.004790] Lustre: lustre-OST2710: new disk, initializing [ 9009.008270] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9009.031056] Lustre: 299221:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9009.155513] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9014.865783] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9014.878730] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9015.017388] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9015.286201] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9024.327514] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9031.423247] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9031.625710] Lustre: lustre-OST03e8: new disk, initializing [ 9031.629221] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9031.634398] Lustre: 300287:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9031.735432] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9038.199424] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9041.004943] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9041.020711] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9041.105355] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9045.635305] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9053.242352] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9053.455278] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9058.106535] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9058.299210] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9062.153702] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9062.344336] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9066.581487] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9066.774625] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9072.101458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9072.103142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9072.121374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9072.130736] Lustre: Skipped 3 previous similar messages [ 9072.143310] Lustre: Skipped 3 previous similar messages [ 9073.888501] Lustre: server umount lustre-MDT0000 complete [ 9077.122372] LustreError: 297245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628289 with bad export cookie 7984003848365974077 [ 9077.126561] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9077.139478] LustreError: 297245:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9077.224384] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9077.229621] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9077.237118] Lustre: Skipped 1 previous similar message [ 9077.254869] Lustre: Skipped 1 previous similar message [ 9077.514711] Lustre: server umount lustre-MDT0001 complete [ 9083.583273] Lustre: server umount lustre-OST2710 complete [ 9088.192654] Lustre: server umount lustre-OST03e8 complete [ 9109.581132] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [ 9119.944559] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9156.798866] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9157.044389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9157.073841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9157.222670] Lustre: lustre-MDT0000: new disk, initializing [ 9157.313592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9157.351369] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9161.694502] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9172.394891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9172.532065] Lustre: 305533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9172.594037] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9172.600331] Lustre: Skipped 1 previous similar message [ 9172.694974] Lustre: lustre-MDT0001: new disk, initializing [ 9172.807976] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9172.845412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9172.854836] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9176.980548] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9186.486255] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9192.350187] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9199.595599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9199.905343] Lustre: lustre-OST0000: new disk, initializing [ 9199.913598] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9199.927311] Lustre: 307503:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9200.068169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9205.852261] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9206.320767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9206.340655] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9206.439582] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9216.026952] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9220.873284] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9221.084412] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9225.755731] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9225.956943] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9232.360943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9232.365334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9232.367772] LustreError: Skipped 1 previous similar message [ 9232.386507] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9232.399437] Lustre: Skipped 1 previous similar message [ 9234.402814] Lustre: server umount lustre-OST0000 complete [ 9239.528170] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9239.540438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9239.560858] Lustre: Skipped 1 previous similar message [ 9239.564363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9244.502422] Lustre: server umount lustre-MDT0000 complete [ 9247.712456] LustreError: 305543:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9247.727341] LustreError: 305543:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9249.086737] LustreError: 307502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628461 with bad export cookie 7984003848365975757 [ 9249.105877] LustreError: 307502:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9249.109911] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9249.499700] Lustre: server umount lustre-MDT0001 complete [ 9259.491856] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:14:29 (1788628469) [ 9261.236430] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9263.558511] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:14:33 (1788628473) [ 9268.249662] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9298.043694] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9298.423825] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9298.456511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9298.635599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9303.802220] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9312.959193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9313.044303] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9313.087535] Lustre: 311360:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9318.489341] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9340.854380] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9341.174243] Lustre: Denying initial registration attempt from nid 192.168.203.109@tcp, specified as failover [ 9341.193990] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9341.204615] LustreError: 312813:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9341.222142] LustreError: 312813:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9341.229685] LustreError: 312813:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9341.359811] Lustre: server umount lustre-OST0000 complete [ 9341.366692] LustreError: 312813:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9349.606715] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9349.617650] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9349.634588] Lustre: Skipped 1 previous similar message [ 9349.650528] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9349.660170] Lustre: Skipped 2 previous similar messages [ 9354.523333] Lustre: server umount lustre-MDT0000 complete [ 9355.754254] LustreError: 312183:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9355.775079] LustreError: 312183:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9357.930635] LustreError: 311353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628569 with bad export cookie 7984003848365976639 [ 9357.934347] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9357.939168] LustreError: 311353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9358.207280] Lustre: server umount lustre-MDT0001 complete [ 9371.691524] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 9376.254364] Key type lgssc unregistered [ 9376.679572] LNet: 313933:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9376.701207] LNetError: 313933:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9376.742068] LNet: Removed LNI 192.168.203.109@tcp [ 9378.023292] Key type .llcrypt unregistered [ 9378.027106] Key type ._llcrypt unregistered [ 9389.457463] Key type ._llcrypt registered [ 9389.464388] Key type .llcrypt registered [ 9389.583037] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9390.349995] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9390.375342] alg: No test for adler32 (adler32-zlib) [ 9391.504647] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 9391.820776] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 9393.567247] Key type lgssc registered [ 9394.613229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9445.134569] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9454.335697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9454.376225] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9455.832716] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9455.846273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9455.950094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9461.692964] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9470.765579] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9470.882224] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9470.924302] Lustre: 316696:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9471.195185] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9475.294480] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9481.791824] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9487.725578] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9507.172544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9507.350761] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9507.516431] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9513.280364] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9520.115812] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9523.692560] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9523.703021] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9523.706644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9523.708696] Lustre: Skipped 1 previous similar message [ 9523.728565] LustreError: Skipped 1 previous similar message [ 9528.361556] Lustre: server umount lustre-OST0000 complete [ 9532.897910] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9532.904920] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9532.925742] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9532.939526] Lustre: Skipped 1 previous similar message [ 9533.934827] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9533.955431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9533.958335] Lustre: Skipped 1 previous similar message [ 9538.380718] Lustre: server umount lustre-MDT0000 complete [ 9539.041353] LustreError: 317450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9539.084161] LustreError: 317450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9542.514698] LustreError: 316689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628754 with bad export cookie 2972516162661020924 [ 9542.526203] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9542.535022] LustreError: 316689:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9542.794742] Lustre: server umount lustre-MDT0001 complete [ 9557.587593] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 9562.233869] Key type lgssc unregistered [ 9562.694578] LNet: 320271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9562.707595] LNetError: 320271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9562.723436] LNet: Removed LNI 192.168.203.109@tcp [ 9563.392763] Key type .llcrypt unregistered [ 9563.397170] Key type ._llcrypt unregistered [ 9590.370962] Key type ._llcrypt registered [ 9590.374429] Key type .llcrypt registered [ 9590.522234] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9591.407593] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9591.469070] alg: No test for adler32 (adler32-zlib) [ 9592.722184] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 9593.037330] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 9594.727166] Key type lgssc registered [ 9595.848269] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9603.754143] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9603.781975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9605.203754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9609.291392] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9615.329282] LustreError: 321684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9615.339127] LustreError: 321684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9616.998772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9617.575861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9623.638571] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9632.948489] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9639.217712] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9647.867309] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9648.270476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9654.913501] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9662.548531] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9673.190650] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9673.192745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9673.214291] Lustre: Skipped 2 previous similar messages [ 9673.232098] Lustre: Skipped 2 previous similar messages [ 9675.641389] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) [ 9677.162852] Lustre: server umount lustre-MDT0000 complete [ 9678.310647] LustreError: 321679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9678.338609] LustreError: 321679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9680.742251] LustreError: 321680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9680.953374] LustreError: 321663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628892 with bad export cookie 1105496993083913557 [ 9680.960909] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9680.971861] LustreError: 321663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9683.424529] LustreError: 324327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9683.424851] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9683.480646] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9685.897066] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.9@tcp (stopping) [ 9685.908299] Lustre: Skipped 1 previous similar message [ 9687.377771] Lustre: server umount lustre-MDT0001 complete [ 9694.504188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9702.531099] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9706.911794] LustreError: 320992:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bb6ee397 x1875513237875712/t0(0) o250->MGC192.168.203.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9707.210819] LustreError: 323517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9707.226983] LustreError: 323517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9707.284697] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9707.340821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9711.916269] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9715.491071] LustreError: 323518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9715.506099] LustreError: 323518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9715.514400] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9719.988080] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9720.238459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9724.234260] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9725.415246] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9731.771398] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9733.123174] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9738.949101] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9741.005520] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9748.449633] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9748.463198] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9748.481415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9751.044863] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9751.049666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9753.795513] Lustre: server umount lustre-OST0000 complete [ 9761.255754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9761.261430] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9761.301525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9761.303641] Lustre: Skipped 2 previous similar messages [ 9773.023296] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9773.177089] Lustre: server umount lustre-MDT0000 complete [ 9776.616042] LustreError: 326014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9776.637244] LustreError: 326014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9776.835056] LustreError: 321664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788628988 with bad export cookie 1105496993083921271 [ 9776.841289] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9776.846859] LustreError: 321664:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9777.160740] Lustre: server umount lustre-MDT0001 complete [ 9788.329724] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [ 9791.003942] Key type lgssc unregistered [ 9791.273827] LNet: 327955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9791.282502] LNetError: 327955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9791.316304] LNet: Removed LNI 192.168.203.109@tcp [ 9792.143432] Key type .llcrypt unregistered [ 9792.145335] Key type ._llcrypt unregistered [ 9816.548516] Key type ._llcrypt registered [ 9816.550959] Key type .llcrypt registered [ 9816.733473] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [ 9817.745738] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9817.797742] alg: No test for adler32 (adler32-zlib) [ 9818.912725] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [ 9819.159534] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [ 9820.871541] Key type lgssc registered [ 9822.107591] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9829.580371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9829.607124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9831.062656] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9835.309833] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9841.640572] LustreError: 329368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9841.668419] LustreError: 329368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9842.998934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.309310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9847.410522] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9854.400588] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9859.396614] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9863.648373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9863.662458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9863.674030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9867.234623] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9867.235524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9867.263725] Lustre: Skipped 1 previous similar message [ 9868.322482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9868.336573] Lustre: Skipped 1 previous similar message [ 9868.607351] Lustre: server umount lustre-MDT0000 complete [ 9872.068267] LustreError: 330984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629084 with bad export cookie 7369054153585873304 [ 9872.071132] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9872.077260] LustreError: 330984:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9872.282085] Lustre: server umount lustre-MDT0001 complete [ 9885.157073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9885.366449] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9885.393871] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9885.581208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9890.311869] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9898.237623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.370041] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9898.403173] Lustre: 331957:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9898.663094] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9902.424388] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9909.006737] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9914.493727] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9923.509486] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9923.727940] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9923.739379] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9923.754172] LustreError: 333839:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 9923.765534] LustreError: 333839:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9923.778057] LustreError: 333839:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9923.887186] Lustre: server umount lustre-OST0000 complete [ 9923.893077] LustreError: 333839:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9934.512375] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9934.749739] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9934.899209] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9941.356534] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9942.117939] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9948.970882] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9956.457973] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9956.570079] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9956.658393] Lustre: lustre-OST0001: new disk, initializing [ 9956.662603] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9956.671494] Lustre: 335380:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9956.774138] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9962.949925] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [ 9964.664466] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9964.673588] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9964.777345] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9969.281404] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9975.270754] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9975.272149] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9975.287835] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9975.291397] LustreError: Skipped 1 previous similar message [ 9975.298743] Lustre: Skipped 1 previous similar message [ 9978.035986] Lustre: server umount lustre-OST0001 complete [ 9980.389575] LustreError: 334725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9980.418668] LustreError: 334725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9982.946160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9982.960356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9982.984277] Lustre: Skipped 1 previous similar message [ 9983.003259] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9985.508046] LustreError: 334725:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9985.516298] LustreError: 334725:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9990.644680] LustreError: 334308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9990.657271] LustreError: 334308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9995.749337] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9995.751986] LustreError: 334308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9995.760399] Lustre: Skipped 6 previous similar messages [ 9995.787387] LustreError: 334308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9996.255200] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9996.427530] Lustre: server umount lustre-OST0000 complete [10000.867687] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10000.868631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10000.872099] LustreError: Skipped 1 previous similar message [10000.892928] Lustre: Skipped 2 previous similar messages [10006.976115] Lustre: server umount lustre-MDT0000 complete [10011.009478] LustreError: 332696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629223 with bad export cookie 7369054153585873857 [10011.017668] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10011.442627] Lustre: server umount lustre-MDT0001 complete [10020.673370] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [10023.123349] Key type lgssc unregistered [10023.359549] LNet: 337337:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10023.365831] LNetError: 337337:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10023.381335] LNet: Removed LNI 192.168.203.109@tcp [10024.150831] Key type .llcrypt unregistered [10024.153257] Key type ._llcrypt unregistered [10091.519283] Key type ._llcrypt registered [10091.525215] Key type .llcrypt registered [10091.649472] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [10107.236810] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10108.503456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10108.574205] alg: No test for adler32 (adler32-zlib) [10109.654819] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [10109.908358] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [10111.617082] Key type lgssc registered [10112.746844] Lustre: Echo OBD driver; http://www.lustre.org/ [10144.361707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10144.425498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10145.726335] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10145.766800] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10145.853361] Lustre: lustre-MDT0000: new disk, initializing [10145.932152] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10145.965206] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10150.311803] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10160.572900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10160.812950] Lustre: 342267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10160.882893] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10160.887127] Lustre: Skipped 1 previous similar message [10160.973855] Lustre: lustre-MDT0001: new disk, initializing [10161.064488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10161.102931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10161.127658] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10165.332989] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10175.183478] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10180.728458] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10187.070446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10187.274750] Lustre: lustre-OST0000: new disk, initializing [10187.282077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10187.290904] Lustre: 344237:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10187.391525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10193.243831] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10193.464110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10193.486415] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10193.601412] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10201.932383] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10206.673197] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10206.925725] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10212.557159] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10212.973632] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10219.489037] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10219.493654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10219.507955] LustreError: Skipped 1 previous similar message [10219.509958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10219.517210] Lustre: Skipped 1 previous similar message [10219.528907] Lustre: Skipped 1 previous similar message [10221.649638] Lustre: server umount lustre-OST0000 complete [10227.680958] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10227.692539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10227.718755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10229.731527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10229.735590] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10231.884280] Lustre: server umount lustre-MDT0000 complete [10235.442209] LustreError: 342257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629447 with bad export cookie 12754162923782314616 [10235.444559] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10235.450783] LustreError: 342257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10235.846730] Lustre: server umount lustre-MDT0001 complete [10244.744391] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:30:54 (1788629454) [10252.709782] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:31:03 (1788629463) [10260.652895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10261.020705] LustreError: 346991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10261.040879] LustreError: 346991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10261.109158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10265.071892] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10266.594299] LustreError: 346992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10271.719641] LustreError: 346991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10273.160803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10273.497762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10277.948984] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10285.523725] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10291.468876] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10298.000658] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10298.295275] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10303.962800] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10310.430874] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10318.875113] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10320.264237] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10321.845700] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10323.338617] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10324.955875] Lustre: Failing over lustre-MDT0000 [10324.965160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -19 [10324.973236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10324.987779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10324.994307] Lustre: Skipped 1 previous similar message [10325.158355] Lustre: server umount lustre-MDT0000 complete [10327.010050] LustreError: 346991:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10327.026789] LustreError: 346991:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10332.129802] LustreError: 346992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10332.153880] LustreError: 346992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10342.369907] LustreError: 346992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10342.390846] LustreError: 346992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10343.303966] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10343.391402] Lustre: 340489:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788629539/real 1788629539] req@00000000ebe038ee x1875513778967936/t0(0) o400->MGC192.168.203.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788629555 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10343.418293] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10353.861760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10353.908048] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10354.010138] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10357.978604] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10359.273723] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10359.300395] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10359.340256] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10366.016794] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10367.330515] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10370.803414] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10372.442401] Lustre: Failing over lustre-MDT0000 [10372.699893] Lustre: server umount lustre-MDT0000 complete [10374.504111] LustreError: 346988:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10374.519478] LustreError: 346988:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [10374.626846] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10374.642650] Lustre: Skipped 3 previous similar messages [10391.011742] Lustre: 340490:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788629586/real 1788629586] req@000000000962db06 x1875513778988032/t0(0) o400->MGC192.168.203.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788629602 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10391.041761] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10392.085106] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10401.276534] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xb0ffe39acf80a514 [10401.292444] Lustre: MGC192.168.203.109@tcp: Connection restored to 0@lo (at 0@lo) [10401.308497] Lustre: Skipped 2 previous similar messages [10401.533790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10401.586065] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10405.211935] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10406.761573] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10406.894084] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10406.942131] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [10406.994900] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10417.977797] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10419.921898] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10427.360930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10427.365288] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10427.365986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10427.369235] Lustre: Skipped 3 previous similar messages [10427.383185] Lustre: Skipped 2 previous similar messages [10432.593410] Lustre: server umount lustre-MDT0000 complete [10435.595937] LustreError: 346971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629647 with bad export cookie 12754162923782317332 [10435.600456] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10435.600836] LustreError: 346971:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10435.781303] Lustre: server umount lustre-MDT0001 complete [10448.403668] Lustre: server umount lustre-OST0000 complete [10464.190729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10464.696230] LustreError: 353334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10464.715113] LustreError: 353334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 27 previous similar messages [10468.577399] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10475.874700] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10476.181708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10476.187783] Lustre: Skipped 1 previous similar message [10479.723847] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10485.889883] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10491.420199] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10497.813623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10503.470397] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10509.426716] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10510.574448] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10516.961451] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10516.969110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10516.978288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10516.984184] Lustre: Skipped 3 previous similar messages [10522.633891] Lustre: server umount lustre-OST0000 complete [10535.029297] Lustre: server umount lustre-MDT0000 complete [10535.424295] LustreError: 353334:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10535.475991] LustreError: 353334:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10540.008803] LustreError: 353314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629752 with bad export cookie 12754162923782318067 [10540.016889] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10540.024196] LustreError: 353314:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10540.330672] Lustre: server umount lustre-MDT0001 complete [10551.002174] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [10554.675284] Key type lgssc unregistered [10555.033785] LNet: 356864:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10555.050959] LNetError: 356864:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10555.078629] LNet: Removed LNI 192.168.203.109@tcp [10556.333757] Key type .llcrypt unregistered [10556.336932] Key type ._llcrypt unregistered [10582.463577] Key type ._llcrypt registered [10582.468488] Key type .llcrypt registered [10582.590210] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10584.080832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10584.112017] alg: No test for adler32 (adler32-zlib) [10585.190649] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [10585.530688] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [10587.319289] Key type lgssc registered [10589.182302] Lustre: Echo OBD driver; http://www.lustre.org/ [10598.858817] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10598.888740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10600.488652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10605.587363] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10610.661970] LustreError: 358378:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10610.677185] LustreError: 358378:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10614.517519] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10615.015035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10619.859429] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10627.407602] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10632.900301] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10638.776327] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10639.080550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10644.856632] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10645.303318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10651.407165] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10661.345229] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10661.354048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10661.366348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10665.953716] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10665.956072] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10665.969547] Lustre: Skipped 1 previous similar message [10665.999818] Lustre: Skipped 1 previous similar message [10667.511185] Lustre: server umount lustre-MDT0000 complete [10670.504823] LustreError: 358358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629882 with bad export cookie 6114765631719328699 [10670.506497] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10670.515132] LustreError: 358358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10670.801158] Lustre: server umount lustre-MDT0001 complete [10684.245595] Lustre: server umount lustre-OST0000 complete [10696.439227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10696.846528] LustreError: 362088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10696.909931] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10700.760632] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10702.311515] LustreError: 362089:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10706.408612] LustreError: 362088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10709.216617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10709.535700] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10713.601741] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10722.498822] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10731.377634] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10739.593369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10739.826512] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10741.869105] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10745.929104] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10753.184249] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10765.793441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10765.800847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10765.822280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10765.824713] Lustre: Skipped 1 previous similar message [10769.893648] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10769.895660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10769.910986] Lustre: Skipped 1 previous similar message [10769.917807] Lustre: Skipped 1 previous similar message [10771.211141] Lustre: server umount lustre-MDT0000 complete [10774.519408] LustreError: 362822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788629986 with bad export cookie 6114765631719329812 [10774.530845] LustreError: 362822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10774.532336] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10774.878993] Lustre: server umount lustre-MDT0001 complete [10789.026888] Lustre: server umount lustre-OST0000 complete [10802.374767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10802.644692] LustreError: 366000:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10802.692243] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10806.227114] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10812.898925] LustreError: 366001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10812.909411] LustreError: 366001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10813.548867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10817.901718] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10825.317676] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10832.620593] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10839.817205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10840.596024] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10840.613445] Lustre: Skipped 1 previous similar message [10847.543729] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [10847.730890] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10855.609598] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10863.078923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10863.087391] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10863.099533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10863.104897] Lustre: Skipped 1 previous similar message [10865.955480] Lustre: server umount lustre-OST0000 complete [10874.336219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10874.345354] Lustre: Skipped 5 previous similar messages [10875.486857] Lustre: server umount lustre-MDT0000 complete [10879.064266] LustreError: 365981:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630091 with bad export cookie 6114765631719330834 [10879.074515] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10879.414612] Lustre: server umount lustre-MDT0001 complete [10888.160328] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [10890.370364] Key type lgssc unregistered [10890.574964] LNet: 369528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10890.586895] LNetError: 369528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10890.601696] LNet: Removed LNI 192.168.203.109@tcp [10891.202565] Key type .llcrypt unregistered [10891.204577] Key type ._llcrypt unregistered [10911.414668] Key type ._llcrypt registered [10911.417452] Key type .llcrypt registered [10911.501699] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10912.049478] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10912.110264] alg: No test for adler32 (adler32-zlib) [10913.094518] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [10913.269164] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [10914.935178] Key type lgssc registered [10915.766916] Lustre: Echo OBD driver; http://www.lustre.org/ [10922.043254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10922.080384] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10922.090864] LustreError: 370944:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10922.121533] LustreError: 370944:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10922.134267] LustreError: 370944:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10922.139176] LustreError: 370944:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10922.153348] LustreError: 370944:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10929.229939] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10929.237424] LustreError: 371341:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10929.253894] LustreError: 371341:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10929.266920] LustreError: 371341:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10929.271157] LustreError: 371341:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10929.281153] LustreError: 371341:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10944.842317] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [10946.816439] Key type lgssc unregistered [10947.040863] LNet: 372344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10947.052148] LNetError: 372344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10947.069041] LNet: Removed LNI 192.168.203.109@tcp [10947.693567] Key type .llcrypt unregistered [10947.698755] Key type ._llcrypt unregistered [10963.668136] Key type ._llcrypt registered [10963.671343] Key type .llcrypt registered [10963.806983] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [10977.092772] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [10978.177427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10978.197643] alg: No test for adler32 (adler32-zlib) [10979.277279] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [10979.553566] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [10981.231155] Key type lgssc registered [10982.256613] Lustre: Echo OBD driver; http://www.lustre.org/ [11011.371512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11011.420075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11012.676564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11012.709327] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11012.846816] Lustre: lustre-MDT0000: new disk, initializing [11012.919943] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11012.965237] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11017.163811] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11025.399554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11025.550905] Lustre: 375490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11025.616171] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11025.619672] Lustre: Skipped 1 previous similar message [11025.703828] Lustre: lustre-MDT0001: new disk, initializing [11025.792062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11025.834022] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11025.849603] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11029.765326] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11038.263434] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11043.496870] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11049.721811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11049.932806] Lustre: lustre-OST0000: new disk, initializing [11049.937510] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11049.945740] Lustre: 377455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11050.043367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11055.330405] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11057.206724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11057.216916] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11057.331792] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11063.275592] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11067.190620] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11067.367267] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11071.465407] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11071.676736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11077.599975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11077.616458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11077.642835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11078.123088] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11078.134789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11078.147145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11078.151595] Lustre: Skipped 1 previous similar message [11079.847743] Lustre: server umount lustre-OST0000 complete [11087.330337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11087.336442] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11087.349091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11093.475736] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11093.484717] Lustre: Skipped 3 previous similar messages [11098.079163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11098.260922] Lustre: server umount lustre-MDT0000 complete [11101.890228] LustreError: 375483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630313 with bad export cookie 2343951472443789781 [11101.905402] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11101.913179] LustreError: 375483:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11102.330531] Lustre: server umount lustre-MDT0001 complete [11111.332314] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11112.961826] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:45:23 (1788630323) [11119.905578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11120.317973] LustreError: 379919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11120.337895] LustreError: 379919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11120.415823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11124.387407] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11125.730387] LustreError: 379920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11130.850559] LustreError: 379919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11132.550790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11132.898087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11137.043677] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11145.044749] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11151.169461] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11158.705577] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11158.998591] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11164.757781] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11171.481214] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11178.834342] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11179.016458] Lustre: lustre-OST0001: new disk, initializing [11179.020567] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11179.027924] Lustre: 382754:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11179.121725] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11186.784240] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11186.804165] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11186.817528] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11186.908982] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11196.774320] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11202.536409] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11202.544033] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11202.552652] Lustre: Skipped 2 previous similar messages [11202.556969] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11206.710585] Lustre: server umount lustre-OST0001 complete [11207.659862] LustreError: 383112:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11207.700094] LustreError: 383112:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11212.774838] LustreError: 383476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11212.791850] LustreError: 383476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11223.014650] LustreError: 383476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11223.030247] LustreError: 383476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11226.593511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11226.602784] LustreError: Skipped 1 previous similar message [11226.608634] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11226.623608] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11226.629565] Lustre: Skipped 2 previous similar messages [11238.367331] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11238.572429] Lustre: server umount lustre-OST0000 complete [11243.489196] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11243.493267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11243.499329] Lustre: Skipped 2 previous similar messages [11243.508316] Lustre: Skipped 7 previous similar messages [11249.526340] Lustre: server umount lustre-MDT0000 complete [11253.739645] LustreError: 381760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11253.779234] LustreError: 381760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11254.684448] LustreError: 379901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630466 with bad export cookie 2343951472443790656 [11254.689354] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11254.696556] LustreError: 379901:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11261.222591] Lustre: server umount lustre-MDT0001 complete [11273.243547] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [11275.564199] Key type lgssc unregistered [11275.804839] LNet: 384743:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11275.819962] LNetError: 384743:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11275.836950] LNet: Removed LNI 192.168.203.109@tcp [11276.519519] Key type .llcrypt unregistered [11276.522333] Key type ._llcrypt unregistered [11324.853828] Key type ._llcrypt registered [11324.859051] Key type .llcrypt registered [11325.019263] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [11326.321136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11326.400407] alg: No test for adler32 (adler32-zlib) [11327.524653] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [11327.907037] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [11329.575188] Key type lgssc registered [11330.783250] Lustre: Echo OBD driver; http://www.lustre.org/ [11338.739799] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11338.786863] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11340.110073] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11340.142539] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11340.377230] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11346.233347] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11355.634052] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11355.728403] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11355.759199] Lustre: 387557:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11356.044505] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11360.651252] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11368.949476] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11376.044039] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11380.706419] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11380.736346] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11380.745398] Lustre: Skipped 1 previous similar message [11384.760573] Lustre: server umount lustre-MDT0000 complete [11388.887413] LustreError: 387550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630600 with bad export cookie 4002393568973871212 [11388.907287] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11388.909339] LustreError: 387550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11389.323673] Lustre: server umount lustre-MDT0001 complete [11400.737250] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:50:10 (1788630610) [11408.861189] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11409.488834] LustreError: 390070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11409.504635] LustreError: 390070:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11409.575184] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11414.963452] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11415.008076] LustreError: 390071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11420.132692] LustreError: 390070:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11425.256106] LustreError: 390071:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11425.651447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11426.041773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11431.254702] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11440.037590] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11446.970780] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11455.270881] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11455.618596] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11455.888558] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11463.154284] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11470.669958] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11477.917492] Lustre: Permanently deactivating lustre-OST0000 [11477.925736] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11487.822700] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11489.239272] LustreError: 393106:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11489.254379] LustreError: 393106:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11489.262597] LustreError: 393106:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11492.696939] LustreError: 393154:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11492.703163] LustreError: 393154:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11501.080711] Lustre: server umount lustre-OST0000 complete [11504.676775] LustreError: 393405:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11504.689503] LustreError: 393405:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11508.195509] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11508.208493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11508.231033] Lustre: Skipped 1 previous similar message [11508.240478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11510.758879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11510.760392] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11515.881198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11515.894562] Lustre: Skipped 2 previous similar messages [11517.098377] Lustre: server umount lustre-MDT0000 complete [11520.520604] LustreError: 390051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630732 with bad export cookie 4002393568973871786 [11520.533901] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11530.969794] Lustre: server umount lustre-MDT0001 complete [11538.346214] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11542.782425] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11545.394452] LustreError: 394562:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11551.595335] Lustre: 394901:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.109@tcp [11553.536599] Lustre: server umount MGS complete [11564.418505] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11564.984452] LustreError: 395625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11565.009072] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11565.015639] Lustre: Skipped 1 previous similar message [11565.070272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11569.695983] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11575.265981] LustreError: 395625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11575.283298] LustreError: 395625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11578.936822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11579.206131] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11584.277716] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11592.187082] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11598.583724] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11600.607212] Lustre: Permanently reactivating lustre-OST0000 [11600.626477] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11600.637338] Lustre: Skipped 2 previous similar messages [11626.975282] LustreError: 397361:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11636.278986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11636.711575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11636.718123] Lustre: Skipped 1 previous similar message [11643.901982] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11654.052705] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11662.139723] Lustre: server umount lustre-OST0000 complete [11666.400076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11666.411985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11666.430359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11678.687360] Lustre: 386876:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788630874/real 1788630874] req@00000000a19a9fb3 x1875515056162816/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788630890 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11678.692567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11678.725637] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11678.725644] Lustre: Skipped 1 previous similar message [11678.742910] Lustre: Skipped 5 previous similar messages [11679.647154] Lustre: 386878:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788630875/real 1788630875] req@0000000052d995c0 x1875515056163584/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788630891 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11680.223130] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11680.459872] Lustre: server umount lustre-MDT0000 complete [11683.826127] LustreError: 395625:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11683.860975] LustreError: 395625:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11684.305107] LustreError: 395605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788630896 with bad export cookie 4002393568973872843 [11684.306534] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11684.321281] LustreError: 395605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11684.710903] Lustre: server umount lustre-MDT0001 complete [11695.509429] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [11698.552634] Key type lgssc unregistered [11698.909677] LNet: 399992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11698.924926] LNetError: 399992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11698.964700] LNet: Removed LNI 192.168.203.109@tcp [11699.849892] Key type .llcrypt unregistered [11699.851785] Key type ._llcrypt unregistered [11719.378567] Key type ._llcrypt registered [11719.381820] Key type .llcrypt registered [11719.465257] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [11735.672939] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [11736.988974] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11737.081811] alg: No test for adler32 (adler32-zlib) [11738.508513] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [11738.852467] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [11740.616840] Key type lgssc registered [11741.790725] Lustre: Echo OBD driver; http://www.lustre.org/ [11781.927570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11781.951708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11783.225101] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11783.296069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11783.378576] Lustre: lustre-MDT0000: new disk, initializing [11783.523779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11783.570209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11788.968234] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11800.101048] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11800.236970] Lustre: 403146:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11800.314762] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11800.319437] Lustre: Skipped 1 previous similar message [11800.482597] Lustre: lustre-MDT0001: new disk, initializing [11800.603875] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11800.649456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11800.675823] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11805.983537] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11816.194715] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11822.253659] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11828.400961] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11828.749610] Lustre: lustre-OST0000: new disk, initializing [11828.752815] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11828.757589] Lustre: 405113:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11828.839607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11835.259394] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11838.086706] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11838.096831] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11838.250889] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11844.567090] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11849.081970] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11849.271333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11853.036346] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11853.398977] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11858.916744] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11858.925507] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11858.926986] LustreError: Skipped 1 previous similar message [11858.937913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11858.952614] Lustre: Skipped 1 previous similar message [11858.965908] Lustre: Skipped 1 previous similar message [11861.544754] Lustre: server umount lustre-OST0000 complete [11867.114908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11867.136504] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11867.168379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11869.155103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11869.157206] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11872.157402] Lustre: server umount lustre-MDT0000 complete [11877.192447] LustreError: 403136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631089 with bad export cookie 14905587504228232899 [11877.196493] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11877.211871] LustreError: 403136:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11877.936502] Lustre: server umount lustre-MDT0001 complete [11889.605385] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11891.387034] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:58:21 (1788631101) [11898.327252] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11898.893132] LustreError: 407574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11898.918233] LustreError: 407574:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11898.984680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11903.980375] LustreError: 407575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11904.422704] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11908.072254] LustreError: 407574:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11913.186482] LustreError: 407575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11913.626049] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11914.119314] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11918.413750] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11926.894767] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11932.711206] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11938.688751] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11938.965381] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11944.675702] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [11952.275763] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11954.498375] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11957.152030] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11963.874212] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11963.883130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11963.886339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11963.892701] Lustre: Skipped 1 previous similar message [11963.909311] Lustre: Skipped 2 previous similar messages [11968.301660] Lustre: server umount lustre-OST0000 complete [11974.120799] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11974.123847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11974.142640] Lustre: Skipped 1 previous similar message [11974.159732] Lustre: Skipped 1 previous similar message [11983.331856] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11983.340140] Lustre: Skipped 3 previous similar messages [11986.399698] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11986.551346] Lustre: server umount lustre-MDT0000 complete [11988.448141] LustreError: 407575:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11988.464815] LustreError: 407575:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11990.080603] LustreError: 407555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631202 with bad export cookie 14905587504228233781 [11990.084567] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11990.088369] LustreError: 407555:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11990.423963] Lustre: server umount lustre-MDT0001 complete [12001.867937] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12004.534388] Key type lgssc unregistered [12004.897270] LNet: 411300:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12004.912122] LNetError: 411300:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12004.921842] LNet: Removed LNI 192.168.203.109@tcp [12005.602176] Key type .llcrypt unregistered [12005.603634] Key type ._llcrypt unregistered [12029.390647] Key type ._llcrypt registered [12029.393766] Key type .llcrypt registered [12029.503628] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12029.822589] Key type .llcrypt unregistered [12029.828518] Key type ._llcrypt unregistered [12047.753656] Key type ._llcrypt registered [12047.757383] Key type .llcrypt registered [12047.880390] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12049.131592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12049.157387] alg: No test for adler32 (adler32-zlib) [12050.236747] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [12050.566438] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [12052.303754] Key type lgssc registered [12053.510460] Lustre: Echo OBD driver; http://www.lustre.org/ [12062.531451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12062.561150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12063.938927] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12069.343186] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12073.445476] LustreError: 413517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12073.458124] LustreError: 413517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12078.063328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12078.463287] LustreError: 413518:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12078.489812] LustreError: 413518:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12078.492401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12083.684956] LustreError: 413517:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12083.727060] LustreError: 413517:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12084.754847] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12088.807946] LustreError: 413512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12088.823504] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12088.828542] LustreError: 413512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12092.564783] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12093.922819] LustreError: 413514:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12093.948826] LustreError: 413514:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12100.283148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12100.709306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12105.655882] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12114.400843] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12114.414087] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12114.441903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12115.937217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12115.947306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12115.951387] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12115.968569] Lustre: Skipped 1 previous similar message [12116.298613] Lustre: server umount lustre-OST0000 complete [12121.057905] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12121.068566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12121.084180] Lustre: Skipped 1 previous similar message [12122.299664] Lustre: server umount lustre-MDT0000 complete [12126.103689] LustreError: 413498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631338 with bad export cookie 3881935129245213623 [12126.106730] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12126.113137] LustreError: 413498:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12126.450377] Lustre: server umount lustre-MDT0001 complete [12138.148256] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12141.114619] Key type lgssc unregistered [12141.411901] LNet: 416666:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12141.421618] LNetError: 416666:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12142.446732] LNet: Removed LNI 192.168.203.109@tcp [12143.536939] Key type .llcrypt unregistered [12143.540177] Key type ._llcrypt unregistered [12171.001702] Key type ._llcrypt registered [12171.004383] Key type .llcrypt registered [12171.182213] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12171.831949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12171.941824] alg: No test for adler32 (adler32-zlib) [12172.993831] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [12173.202292] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [12174.855166] Key type lgssc registered [12176.072780] Lustre: Echo OBD driver; http://www.lustre.org/ [12183.819126] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12183.844708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12199.327389] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12204.447695] LustreError: 418051:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.109@tcp: failed processing log, type 4: rc = -110 [12231.071516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12236.502759] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12243.301466] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12251.087824] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12251.686276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12251.752985] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12256.691539] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12256.741499] LustreError: 418076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12256.770237] LustreError: 418076:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12261.862482] LustreError: 418077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12264.951779] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12265.200494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12269.647131] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12276.339737] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12281.726558] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12290.798191] Lustre: server umount lustre-OST0000 complete [12296.160633] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12296.165829] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12296.174890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12297.189077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12297.190148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12302.305684] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12302.318567] Lustre: Skipped 2 previous similar messages [12304.443418] Lustre: server umount lustre-MDT0000 complete [12307.424142] LustreError: 419093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12307.425743] Lustre: 417391:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788631503/real 1788631503] req@0000000060976c41 x1875515943187328/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788631519 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12307.452253] LustreError: 419093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12307.494389] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12308.410926] LustreError: 418057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631520 with bad export cookie 18110438949311564220 [12308.424201] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12308.429414] LustreError: 418057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12308.774981] Lustre: server umount lustre-MDT0001 complete [12319.083091] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12321.693748] Key type lgssc unregistered [12321.971076] LNet: 421624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12321.977762] LNetError: 421624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12322.003855] LNet: Removed LNI 192.168.203.109@tcp [12322.886903] Key type .llcrypt unregistered [12322.894593] Key type ._llcrypt unregistered [12344.805731] Key type ._llcrypt registered [12344.808181] Key type .llcrypt registered [12344.897381] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12345.930467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12345.952510] alg: No test for adler32 (adler32-zlib) [12347.005944] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [12347.211854] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [12348.927176] Key type lgssc registered [12350.186665] Lustre: Echo OBD driver; http://www.lustre.org/ [12357.456212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12357.481185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12358.919744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12362.790125] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12368.359295] LustreError: 423017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12368.373569] LustreError: 423017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12370.921756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12371.402107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12375.665461] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12383.470918] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12388.933040] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12396.932494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12397.363504] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12403.519521] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12404.908990] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12412.428813] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12417.189629] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12417.195968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12417.218792] Lustre: Skipped 2 previous similar messages [12417.233810] Lustre: Skipped 2 previous similar messages [12419.495292] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.9@tcp (stopping) [12422.416949] Lustre: server umount lustre-MDT0000 complete [12422.628156] LustreError: 425372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12422.642574] LustreError: 425372:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12424.538383] LustreError: 423013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12425.023265] LustreError: 422997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.9@tcp arrived at 1788631637 with bad export cookie 10821818711021324074 [12425.042606] LustreError: 422997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12427.020363] LustreError: 425712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788631639 with bad export cookie 10821818711021323185 [12427.044749] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12427.746116] LustreError: 423011:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12427.758477] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12427.777159] LustreError: 423011:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12427.799578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12432.866692] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12432.867346] LustreError: 423745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12432.900800] LustreError: 423745:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12433.342132] Lustre: server umount lustre-OST0000 complete [12441.824379] LustreError: 423747:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000a542e803 x1875516125643136/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12441.983432] Lustre: server umount lustre-MDT0001 complete [12453.518858] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12456.332085] Key type lgssc unregistered [12456.592173] LNet: 426704:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12456.603435] LNetError: 426704:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12456.630434] LNet: Removed LNI 192.168.203.109@tcp [12457.403086] Key type .llcrypt unregistered [12457.406733] Key type ._llcrypt unregistered [12484.604406] Key type ._llcrypt registered [12484.610554] Key type .llcrypt registered [12484.830855] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12485.478330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12485.600145] alg: No test for adler32 (adler32-zlib) [12486.762523] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [12487.100175] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [12488.863258] Key type lgssc registered [12489.989745] Lustre: Echo OBD driver; http://www.lustre.org/ [12497.927040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12497.958726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12499.507636] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12503.519600] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12509.665550] LustreError: 428116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12509.679852] LustreError: 428116:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12511.297726] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12511.619986] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12515.904111] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12522.775513] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12527.364569] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12532.906115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12533.215638] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12539.132536] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12545.523308] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12545.938787] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12550.631453] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12550.636260] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12550.648829] Lustre: Skipped 1 previous similar message [12550.655844] Lustre: Skipped 1 previous similar message [12553.593508] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.9@tcp (stopping) [12555.747031] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12555.758504] Lustre: Skipped 1 previous similar message [12556.488320] Lustre: server umount lustre-MDT0001 complete [12560.865255] LustreError: 429154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12560.895205] LustreError: 429154:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12565.992214] LustreError: 428116:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12565.993093] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12566.033070] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12566.049257] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12568.350440] Lustre: server umount lustre-OST0000 complete [12571.109911] LustreError: 429154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12572.467505] Lustre: server umount lustre-MDT0000 complete [12587.592411] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12590.991680] Key type lgssc unregistered [12591.329709] LNet: 431798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12591.352936] LNetError: 431798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12591.377745] LNet: Removed LNI 192.168.203.109@tcp [12592.455864] Key type .llcrypt unregistered [12592.463298] Key type ._llcrypt unregistered [12622.006853] Key type ._llcrypt registered [12622.011761] Key type .llcrypt registered [12622.196192] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [12637.017709] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12638.274252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12638.347082] alg: No test for adler32 (adler32-zlib) [12639.533545] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [12639.925332] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [12641.697127] Key type lgssc registered [12643.703045] Lustre: Echo OBD driver; http://www.lustre.org/ [12682.766702] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12698.399090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12698.453274] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12699.779861] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12699.820869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12699.933550] Lustre: lustre-MDT0000: new disk, initializing [12700.008053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12700.027710] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12704.790674] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12719.814605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12720.228479] Lustre: lustre-OST0000: new disk, initializing [12720.232568] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12720.242086] Lustre: Skipped 1 previous similar message [12720.252806] Lustre: 436827:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12720.355418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12726.879208] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12726.896498] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12726.917633] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12726.931956] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12736.371566] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12752.494832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12752.620329] Lustre: 435811:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12752.682544] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12752.754271] Lustre: lustre-MDT0001: new disk, initializing [12752.876866] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12752.933112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12752.949467] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12757.143938] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12765.677407] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12765.889794] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12769.887637] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12770.093786] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12775.274824] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.9@tcp (stopping) [12777.961480] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12777.975133] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12777.981381] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12777.997084] Lustre: Skipped 1 previous similar message [12778.018716] Lustre: Skipped 1 previous similar message [12780.689981] Lustre: server umount lustre-MDT0001 complete [12785.516391] LustreError: 435816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12785.540974] LustreError: 435816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12788.208614] LustreError: 435821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12788.237048] LustreError: 435821:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12793.321359] LustreError: 435817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12793.338516] LustreError: 435817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12798.436741] LustreError: 435818:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12798.451550] LustreError: 435818:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12803.552242] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12803.554370] LustreError: 435821:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12803.567773] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12803.609358] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12806.264835] Lustre: server umount lustre-OST0000 complete [12817.326522] Lustre: server umount lustre-MDT0000 complete [12833.391388] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12836.146849] Key type lgssc unregistered [12836.417857] LNet: 440577:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12836.424748] LNetError: 440577:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12836.443752] LNet: Removed LNI 192.168.203.109@tcp [12837.252962] Key type .llcrypt unregistered [12837.255102] Key type ._llcrypt unregistered [12877.308441] Key type ._llcrypt registered [12877.311512] Key type .llcrypt registered [12877.415826] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [12877.713527] Key type .llcrypt unregistered [12877.717429] Key type ._llcrypt unregistered [12894.047502] Key type ._llcrypt registered [12894.049307] Key type .llcrypt registered [12894.170592] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [12895.327320] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12895.371243] alg: No test for adler32 (adler32-zlib) [12896.407790] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [12896.642964] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [12898.343219] Key type lgssc registered [12899.652931] Lustre: Echo OBD driver; http://www.lustre.org/ [12929.050511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12929.089110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12930.316689] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12930.388797] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12930.532623] Lustre: lustre-MDT0000: new disk, initializing [12930.684452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12930.727091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12935.763943] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12946.501658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12946.661876] Lustre: 443948:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12946.802414] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12946.811684] Lustre: Skipped 1 previous similar message [12946.938395] Lustre: lustre-MDT0001: new disk, initializing [12947.091771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12947.155525] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12947.178952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12951.076461] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12961.092410] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12967.104793] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12973.539341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12973.911640] Lustre: lustre-OST0000: new disk, initializing [12973.916821] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12973.924445] Lustre: 445914:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12974.046476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12981.025291] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [12981.813723] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12981.837645] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12981.951524] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12990.313557] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12997.607868] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12997.616496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12997.618979] Lustre: Skipped 1 previous similar message [12997.631129] Lustre: Skipped 2 previous similar messages [13002.725218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13002.735529] Lustre: Skipped 2 previous similar messages [13003.100534] Lustre: server umount lustre-MDT0000 complete [13006.755818] LustreError: 444742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632218 with bad export cookie 13672409858543869646 [13006.760341] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13006.765759] LustreError: 444742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13007.057449] Lustre: server umount lustre-MDT0001 complete [13021.061692] Lustre: server umount lustre-OST0000 complete [13030.573559] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:17:21 (1788632241) [13038.391389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13039.071023] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13043.756636] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [13044.196148] LustreError: 447945:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13044.234984] LustreError: 447945:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13048.288991] LustreError: 447944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13052.008106] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13052.493430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13057.654428] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [13065.899450] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13073.004799] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13092.465289] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13092.736636] Lustre: Found index 0 for lustre-OST0000, updating log [13092.886883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13099.139454] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [13107.450766] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13109.297991] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13118.433523] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13118.439454] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13118.459926] Lustre: Skipped 1 previous similar message [13118.467538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13123.950490] Lustre: server umount lustre-OST0000 complete [13140.960315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13140.964400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13140.988887] Lustre: Skipped 2 previous similar messages [13140.990592] Lustre: Skipped 3 previous similar messages [13145.125104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13145.129512] Lustre: Skipped 1 previous similar message [13146.133528] Lustre: server umount lustre-MDT0000 complete [13146.157425] LustreError: 447944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13146.172529] LustreError: 447944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13150.256209] LustreError: 447926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788632362 with bad export cookie 13672409858543871046 [13150.266810] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13150.278125] LustreError: 447926:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13150.767909] Lustre: server umount lustre-MDT0001 complete [13159.733069] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:19:30 (1788632370) [13160.863254] LNet: 1 local NIs in recovery (showing 1): 192.168.203.109@tcp [13782.006391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13782.728307] LustreError: 454356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13782.827154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13788.131392] LustreError: 454357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13788.575624] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [13792.226405] LustreError: 454356:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13798.268802] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13798.368958] LustreError: 454357:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13798.392796] LustreError: 454357:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13798.760573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13804.221683] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [13813.023594] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13819.198451] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13838.751347] LNet: 1 local NIs in recovery (showing 1): 192.168.203.109@tcp [13839.987116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13840.337617] Lustre: Found index 0 for lustre-OST0000, updating log [13840.716284] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13847.950864] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [13857.635426] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13861.092687] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13861.121557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13866.303791] Lustre: server umount lustre-OST0000 complete [13871.654060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13871.674532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13871.688038] Lustre: Skipped 1 previous similar message [13873.844586] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13873.847476] Lustre: Skipped 2 previous similar messages [13876.996275] LustreError: 454351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13877.007826] LustreError: 454351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13877.105560] Lustre: server umount lustre-MDT0000 complete [13881.935618] LustreError: 454337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633093 with bad export cookie 13672409858543872131 [13881.950303] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13881.961319] LustreError: 454337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13882.612038] Lustre: server umount lustre-MDT0001 complete [13898.143987] LNet: 1 local NIs in recovery (showing 1): 192.168.203.109@tcp [13915.330698] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [13927.956910] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [13970.876876] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:33:01 (1788633181) [13993.141709] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [13995.817510] Key type lgssc unregistered [13996.128692] LNet: 460970:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13996.145716] LNetError: 460970:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14103.666546] LNet: Removed LNI 192.168.203.109@tcp [14104.785070] Key type .llcrypt unregistered [14104.790193] Key type ._llcrypt unregistered [14123.521189] Key type ._llcrypt registered [14123.524989] Key type .llcrypt registered [14123.644495] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [14124.758283] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14124.821868] alg: No test for adler32 (adler32-zlib) [14126.069150] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [14126.417901] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [14128.111270] Key type lgssc registered [14129.374859] Lustre: Echo OBD driver; http://www.lustre.org/ [14133.973706] LNet: Added LNI 192.168.203.109@tcp100 [8/256/0/180] [14134.938832] LNet: Added LNI 192.168.203.109@tcp101 [8/256/0/180] [14135.816560] LNet: Added LNI 192.168.203.109@tcp102 [8/256/0/180] [14136.682591] LNet: Added LNI 192.168.203.109@tcp103 [8/256/0/180] [14137.508179] LNet: Added LNI 192.168.203.109@tcp104 [8/256/0/180] [14138.338284] LNet: Added LNI 192.168.203.109@tcp105 [8/256/0/180] [14139.275804] LNet: Added LNI 192.168.203.109@tcp106 [8/256/0/180] [14140.056501] LNet: Added LNI 192.168.203.109@tcp107 [8/256/0/180] [14140.892559] LNet: Added LNI 192.168.203.109@tcp108 [8/256/0/180] [14141.671061] LNet: Added LNI 192.168.203.109@tcp109 [8/256/0/180] [14142.460501] LNet: Added LNI 192.168.203.109@tcp110 [8/256/0/180] [14143.392606] LNet: Added LNI 192.168.203.109@tcp111 [8/256/0/180] [14144.341752] LNet: Added LNI 192.168.203.109@tcp112 [8/256/0/180] [14145.236714] LNet: Added LNI 192.168.203.109@tcp113 [8/256/0/180] [14146.412909] LNet: Added LNI 192.168.203.109@tcp114 [8/256/0/180] [14147.526596] LNet: Added LNI 192.168.203.109@tcp115 [8/256/0/180] [14148.537607] LNet: Added LNI 192.168.203.109@tcp116 [8/256/0/180] [14149.481428] LNet: Added LNI 192.168.203.109@tcp117 [8/256/0/180] [14150.344279] LNet: Added LNI 192.168.203.109@tcp118 [8/256/0/180] [14151.224420] LNet: Added LNI 192.168.203.109@tcp119 [8/256/0/180] [14152.271754] LNet: Added LNI 192.168.203.109@tcp120 [8/256/0/180] [14153.113636] LNet: Added LNI 192.168.203.109@tcp121 [8/256/0/180] [14154.052557] LNet: Added LNI 192.168.203.109@tcp122 [8/256/0/180] [14155.191475] LNet: Added LNI 192.168.203.109@tcp123 [8/256/0/180] [14156.358042] LNet: Added LNI 192.168.203.109@tcp124 [8/256/0/180] [14157.353658] LNet: Added LNI 192.168.203.109@tcp125 [8/256/0/180] [14158.343700] LNet: Added LNI 192.168.203.109@tcp126 [8/256/0/180] [14159.209992] LNet: Added LNI 192.168.203.109@tcp127 [8/256/0/180] [14160.249867] LNet: Added LNI 192.168.203.109@tcp128 [8/256/0/180] [14161.139044] LNet: Added LNI 192.168.203.109@tcp129 [8/256/0/180] [14162.105654] LNet: Added LNI 192.168.203.109@tcp130 [8/256/0/180] [14163.129377] LNet: Added LNI 192.168.203.109@tcp131 [8/256/0/180] [14164.105622] LNet: Added LNI 192.168.203.109@tcp132 [8/256/0/180] [14164.964865] LNet: Added LNI 192.168.203.109@tcp133 [8/256/0/180] [14165.884161] LNet: Added LNI 192.168.203.109@tcp134 [8/256/0/180] [14166.944698] LNet: Added LNI 192.168.203.109@tcp135 [8/256/0/180] [14174.944598] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14174.971845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14176.121482] Lustre: 463998:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.109@tcp: can't add NID 192.168.203.109@tcp130: rc = -75 [14176.186397] LNetError: 464020:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.109 [14176.201803] LNetError: Protocol error connecting to 192.168.203.109@tcp119 at host 192.168.203.109:988: is it running a compatible version of Lustre? [14176.230684] LNetError: Skipped 10 previous similar messages [14176.296074] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14176.338537] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14176.444162] Lustre: lustre-MDT0000: new disk, initializing [14176.580982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14176.598262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14176.616574] LNet: 6 local NIs in recovery (showing 6): 192.168.203.109@tcp123, 192.168.203.109@tcp127, 192.168.203.109@tcp128, 192.168.203.109@tcp126, 192.168.203.109@tcp129, 192.168.203.109@tcp119 [14177.251290] LNetError: 464022:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.109 [14177.272432] LNetError: 464022:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14177.286378] LNetError: Protocol error connecting to 192.168.203.109@tcp124 at host 192.168.203.109:988: is it running a compatible version of Lustre? [14179.790878] LNetError: 461455:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.109 [14179.809166] LNetError: 461455:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [14179.833367] LNetError: Protocol error connecting to 192.168.203.109@tcp123 at host 192.168.203.109:988: is it running a compatible version of Lustre? [14179.852735] LNetError: Skipped 6 previous similar messages [14182.110421] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [14183.907679] LNetError: 464017:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.109 [14183.926285] LNetError: Protocol error connecting to 192.168.203.109@tcp123 at host 192.168.203.109:988: is it running a compatible version of Lustre? [14191.090911] LNetError: 464033:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.109 [14191.113049] LNetError: Protocol error connecting to 192.168.203.109@tcp123 at host 192.168.203.109:988: is it running a compatible version of Lustre? [14193.678693] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14193.879652] Lustre: 464015:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14193.956886] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14193.964633] Lustre: Skipped 1 previous similar message [14194.044878] Lustre: lustre-MDT0001: new disk, initializing [14194.092068] Lustre: 464820:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.109@tcp: can't add NID 192.168.203.109@tcp130: rc = -75 [14194.114542] Lustre: 464820:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14194.331464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14194.375560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14194.386905] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14200.074656] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [14200.389217] Lustre: 465089:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.109@tcp: can't add NID 192.168.203.109@tcp130: rc = -75 [14200.418096] Lustre: 465089:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14206.310041] LNetError: 461453:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.109 [14206.319265] LNetError: Protocol error connecting to 192.168.203.109@tcp112 at host 192.168.203.109:988: is it running a compatible version of Lustre? [14211.275589] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14217.917860] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14224.740739] LNetError: 461455:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.109 [14224.757659] LNetError: 461455:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [14224.767744] LNetError: Protocol error connecting to 192.168.203.109@tcp121 at host 192.168.203.109:988: is it running a compatible version of Lustre? [14224.783245] LNetError: Skipped 8 previous similar messages [14225.954779] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14226.264423] Lustre: lustre-OST0000: new disk, initializing [14226.267569] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14226.276306] Lustre: 465993:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14226.390629] Lustre: 465998:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.109@tcp: can't add NID 192.168.203.109@tcp130: rc = -75 [14226.397297] Lustre: 465998:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14226.458784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14234.012573] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [14234.640540] Lustre: 466373:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.109@tcp: can't add NID 192.168.203.109@tcp130: rc = -75 [14234.653380] Lustre: 466373:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14234.760284] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14234.789163] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14234.910246] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14236.127305] LNet: 1 local NIs in recovery (showing 1): 192.168.203.109@tcp123 [14243.931067] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14255.584405] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14255.600563] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14255.621894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14255.633407] Lustre: Skipped 1 previous similar message [14257.608567] Lustre: server umount lustre-OST0000 complete [14264.673370] LNetError: 461454:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.109 [14264.688514] LNetError: 461454:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 19 previous similar messages [14264.701494] LNetError: Protocol error connecting to 192.168.203.109@tcp124 at host 192.168.203.109:988: is it running a compatible version of Lustre? [14264.713162] LNetError: Skipped 19 previous similar messages [14265.826630] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14265.828233] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14265.849566] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14265.861830] LustreError: Skipped 1 previous similar message [14265.878366] Lustre: Skipped 2 previous similar messages [14270.956278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14270.972673] Lustre: Skipped 1 previous similar message [14275.042218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14275.059060] Lustre: Skipped 1 previous similar message [14276.063542] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14276.283500] Lustre: server umount lustre-MDT0000 complete [14280.191482] LustreError: 464007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788633492 with bad export cookie 3154297590955691774 [14280.198713] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14280.207058] LustreError: 464007:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14280.666686] Lustre: server umount lustre-MDT0001 complete [14293.636248] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [14297.024166] Key type lgssc unregistered [14297.559417] LNet: 467907:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14297.567177] LNetError: 467907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14297.579905] LNet: Removed LNI 192.168.203.109@tcp [14298.600382] LNet: Removed LNI 192.168.203.109@tcp100 [14298.603364] LNetError: 467907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14298.610901] LNetError: 467907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14298.615379] LNet: Removed LNI 192.168.203.109@tcp101 [14299.679155] LNet: Removed LNI 192.168.203.109@tcp102 [14299.685854] LNetError: 467907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14299.698838] LNetError: 467907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14299.725872] LNet: Removed LNI 192.168.203.109@tcp103 [14300.767233] LNet: Removed LNI 192.168.203.109@tcp104 [14301.791784] LNet: Removed LNI 192.168.203.109@tcp105 [14301.801496] LNetError: 467907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14301.818766] LNetError: 467907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14301.845968] LNet: Removed LNI 192.168.203.109@tcp106 [14302.879370] LNet: Removed LNI 192.168.203.109@tcp107 [14303.903277] LNet: Removed LNI 192.168.203.109@tcp108 [14304.928325] LNet: Removed LNI 192.168.203.109@tcp109 [14305.951161] LNet: Removed LNI 192.168.203.109@tcp110 [14305.968493] LNetError: 467907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14305.981121] LNetError: 467907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14305.994939] LNet: Removed LNI 192.168.203.109@tcp111 [14307.042485] LNet: Removed LNI 192.168.203.109@tcp112 [14308.063763] LNet: Removed LNI 192.168.203.109@tcp113 [14309.087394] LNet: Removed LNI 192.168.203.109@tcp114 [14310.112085] LNet: Removed LNI 192.168.203.109@tcp115 [14311.135812] LNet: Removed LNI 192.168.203.109@tcp116 [14312.160721] LNet: Removed LNI 192.168.203.109@tcp117 [14313.183197] LNet: Removed LNI 192.168.203.109@tcp118 [14314.207179] LNet: Removed LNI 192.168.203.109@tcp119 [14314.217126] LNetError: 467907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14314.240240] LNetError: 467907:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14314.246530] LNet: Removed LNI 192.168.203.109@tcp120 [14315.295679] LNet: Removed LNI 192.168.203.109@tcp121 [14316.319558] LNet: Removed LNI 192.168.203.109@tcp122 [14316.324911] LNet: Removed LNI 192.168.203.109@tcp123 [14317.343343] LNet: Removed LNI 192.168.203.109@tcp124 [14318.368078] LNet: Removed LNI 192.168.203.109@tcp125 [14319.391254] LNet: Removed LNI 192.168.203.109@tcp126 [14320.415245] LNet: Removed LNI 192.168.203.109@tcp127 [14321.441240] LNet: Removed LNI 192.168.203.109@tcp128 [14322.464728] LNet: Removed LNI 192.168.203.109@tcp129 [14322.474542] LNet: Removed LNI 192.168.203.109@tcp130 [14322.482899] LNet: Removed LNI 192.168.203.109@tcp131 [14322.488224] LNet: Removed LNI 192.168.203.109@tcp132 [14322.498395] LNet: Removed LNI 192.168.203.109@tcp133 [14322.521980] LNet: Removed LNI 192.168.203.109@tcp134 [14322.539223] LNet: Removed LNI 192.168.203.109@tcp135 [14323.685786] Key type .llcrypt unregistered [14323.690327] Key type ._llcrypt unregistered [14352.267306] Key type ._llcrypt registered [14352.269517] Key type .llcrypt registered [14352.354572] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [14371.577841] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [14373.301162] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14373.344654] alg: No test for adler32 (adler32-zlib) [14374.594297] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [14374.902201] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [14376.663538] Key type lgssc registered [14377.987918] Lustre: Echo OBD driver; http://www.lustre.org/ [14419.254550] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 14:40:29 (1788633629) [14441.054961] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [14444.008983] Key type lgssc unregistered [14444.328877] LNet: 471996:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14444.340085] LNetError: 471996:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14444.368756] LNet: Removed LNI 192.168.203.109@tcp [14445.339441] Key type .llcrypt unregistered [14445.345257] Key type ._llcrypt unregistered [14463.159607] Key type ._llcrypt registered [14463.163532] Key type .llcrypt registered [14463.282893] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [14464.883215] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14464.913747] alg: No test for adler32 (adler32-zlib) [14466.085308] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [14466.397531] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [14468.095198] Key type lgssc registered [14469.357579] Lustre: Echo OBD driver; http://www.lustre.org/ [14479.224656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14479.252173] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14480.545492] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14480.584917] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14480.662897] Lustre: lustre-MDT0000: new disk, initializing [14480.787530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14480.815147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14486.145880] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [14497.986659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14498.103231] Lustre: 473204:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14498.153481] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14498.157910] Lustre: Skipped 1 previous similar message [14498.223755] Lustre: lustre-MDT0001: new disk, initializing [14498.303731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14498.346262] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14498.364995] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14503.326099] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [14513.915510] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14521.101737] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14530.492237] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14531.071184] Lustre: lustre-OST0000: new disk, initializing [14531.078847] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14531.104844] Lustre: 475175:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14531.225708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14540.397324] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14540.413469] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14540.654769] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14541.438827] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [14553.375849] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14557.922993] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing lnet_if_list [14563.753785] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14563.793067] LNet: Added LNI 192.168.203.109@tcp10 [8/256/0/180] [14570.243138] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14570.282806] LNet: Added LNI 192.168.203.109@tcp11 [8/256/0/180] [14575.067270] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14575.112548] LNet: Added LNI 192.168.203.109@tcp12 [8/256/0/180] [14580.367043] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14580.391608] LNet: Added LNI 192.168.203.109@tcp13 [8/256/0/180] [14585.558257] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14585.587216] LNet: Added LNI 192.168.203.109@tcp14 [8/256/0/180] [14590.890119] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14590.925079] LNet: Added LNI 192.168.203.109@tcp15 [8/256/0/180] [14596.596698] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14596.671865] LNet: Added LNI 192.168.203.109@tcp16 [8/256/0/180] [14602.574645] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14602.598336] LNet: Added LNI 192.168.203.109@tcp17 [8/256/0/180] [14608.826264] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14608.894070] LNet: Added LNI 192.168.203.109@tcp18 [8/256/0/180] [14615.596926] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14615.629789] LNet: Added LNI 192.168.203.109@tcp19 [8/256/0/180] [14622.421843] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14622.486699] LNet: Added LNI 192.168.203.109@tcp20 [8/256/0/180] [14629.288394] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14629.335292] LNet: Added LNI 192.168.203.109@tcp21 [8/256/0/180] [14636.433341] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14636.488450] LNet: Added LNI 192.168.203.109@tcp22 [8/256/0/180] [14643.803629] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14643.874575] LNet: Added LNI 192.168.203.109@tcp23 [8/256/0/180] [14650.766901] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14650.804586] LNet: Added LNI 192.168.203.109@tcp24 [8/256/0/180] [14657.647529] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14657.706632] LNet: Added LNI 192.168.203.109@tcp25 [8/256/0/180] [14665.020305] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14665.041392] LNet: Added LNI 192.168.203.109@tcp26 [8/256/0/180] [14671.983178] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14672.002243] LNet: Added LNI 192.168.203.109@tcp27 [8/256/0/180] [14679.744608] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14679.787036] LNet: Added LNI 192.168.203.109@tcp28 [8/256/0/180] [14687.213501] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14687.268715] LNet: Added LNI 192.168.203.109@tcp29 [8/256/0/180] [14695.418365] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14695.458841] LNet: Added LNI 192.168.203.109@tcp30 [8/256/0/180] [14703.322787] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14703.348714] LNet: Added LNI 192.168.203.109@tcp31 [8/256/0/180] [14711.691821] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14711.723506] LNet: Added LNI 192.168.203.109@tcp32 [8/256/0/180] [14722.443261] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14722.525297] LNet: Added LNI 192.168.203.109@tcp33 [8/256/0/180] [14731.454321] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14731.513334] LNet: Added LNI 192.168.203.109@tcp34 [8/256/0/180] [14740.016835] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14740.050080] LNet: Added LNI 192.168.203.109@tcp35 [8/256/0/180] [14747.292680] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14747.333503] LNet: Added LNI 192.168.203.109@tcp36 [8/256/0/180] [14755.256702] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14755.288123] LNet: Added LNI 192.168.203.109@tcp37 [8/256/0/180] [14763.268229] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14763.304963] LNet: Added LNI 192.168.203.109@tcp38 [8/256/0/180] [14771.745834] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14771.764616] LNet: Added LNI 192.168.203.109@tcp39 [8/256/0/180] [14778.830038] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14778.852176] LNet: Added LNI 192.168.203.109@tcp40 [8/256/0/180] [14786.991427] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14787.025517] LNet: Added LNI 192.168.203.109@tcp41 [8/256/0/180] [14795.352376] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14795.392494] LNet: Added LNI 192.168.203.109@tcp42 [8/256/0/180] [14803.234116] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14803.256746] LNet: Added LNI 192.168.203.109@tcp43 [8/256/0/180] [14810.687935] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14810.733838] LNet: Added LNI 192.168.203.109@tcp44 [8/256/0/180] [14819.678896] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14819.707199] LNet: Added LNI 192.168.203.109@tcp45 [8/256/0/180] [14828.607778] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14828.657761] LNet: Added LNI 192.168.203.109@tcp46 [8/256/0/180] [14838.459940] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14838.481727] LNet: Added LNI 192.168.203.109@tcp47 [8/256/0/180] [14848.573528] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14848.617372] LNet: Added LNI 192.168.203.109@tcp48 [8/256/0/180] [14857.903863] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14857.971364] LNet: Added LNI 192.168.203.109@tcp49 [8/256/0/180] [14867.110074] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14867.134395] LNet: Added LNI 192.168.203.109@tcp50 [8/256/0/180] [14887.393908] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14887.405558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14887.436493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14888.934287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14888.940312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14888.979329] Lustre: Skipped 2 previous similar messages [14894.053482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14899.167257] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14899.173879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14899.190798] Lustre: Skipped 4 previous similar messages [14899.561082] Lustre: server umount lustre-MDT0000 complete [14903.773929] LustreError: 473197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634115 with bad export cookie 12437447961279043999 [14903.778968] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14903.787724] LustreError: 473197:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14904.152870] Lustre: server umount lustre-MDT0001 complete [14920.672093] Lustre: 472477:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788634116/real 1788634116] req@00000000892eed4f x1875518347733888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788634132 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14920.754856] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14920.782333] Lustre: Skipped 1 previous similar message [14924.767133] Lustre: 472478:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788634120/real 1788634120] req@000000008cb7ec4c x1875518347734016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788634136 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14924.939652] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [14925.861027] Lustre: 472478:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788634121/real 1788634121] req@000000009687b43c x1875518347734144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788634137 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14930.026958] Lustre: 472477:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788634125/real 1788634125] req@00000000abce014c x1875518347734400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788634141 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14934.176210] Lustre: 472477:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788634130/real 1788634130] req@00000000cd291240 x1875518347734528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788634146 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14935.404103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14949.473864] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xac9ab1063c71a442 [14949.494838] Lustre: MGC192.168.203.109@tcp: Connection restored to 0@lo (at 0@lo) [14949.755555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14954.851339] LustreError: 475168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14954.889290] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14954.896637] LustreError: 475168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [14956.177149] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [14960.099723] LustreError: 475169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14965.222869] LustreError: 475168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14965.263390] LustreError: 475168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14968.967807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14969.606021] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14974.971058] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14975.137976] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [14984.110151] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14990.530453] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15015.907909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15015.910477] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15015.931807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15015.946724] Lustre: Skipped 1 previous similar message [15015.957285] Lustre: Skipped 1 previous similar message [15017.940373] Lustre: server umount lustre-OST0000 complete [15026.145468] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15026.147553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15026.161894] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15026.187840] Lustre: Skipped 1 previous similar message [15026.203228] Lustre: Skipped 1 previous similar message [15036.383183] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15036.733645] Lustre: server umount lustre-MDT0000 complete [15040.747645] LustreError: 475174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634252 with bad export cookie 12437447961279046722 [15040.751758] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15040.760056] LustreError: 475174:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15041.329191] Lustre: server umount lustre-MDT0001 complete [15053.211748] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [15056.246428] Key type lgssc unregistered [15056.782858] LNet: 502232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15056.797570] LNetError: 502232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15057.830284] LNet: Removed LNI 192.168.203.109@tcp [15057.833222] LNetError: 502232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15057.845952] LNet: Removed LNI 192.168.203.109@tcp10 [15057.849504] LNet: Removed LNI 192.168.203.109@tcp11 [15057.852774] LNet: Removed LNI 192.168.203.109@tcp12 [15057.855792] LNet: Removed LNI 192.168.203.109@tcp13 [15057.864474] LNet: Removed LNI 192.168.203.109@tcp14 [15057.868953] LNet: Removed LNI 192.168.203.109@tcp15 [15057.873089] LNet: Removed LNI 192.168.203.109@tcp16 [15057.875464] LNet: Removed LNI 192.168.203.109@tcp17 [15057.882690] LNet: Removed LNI 192.168.203.109@tcp18 [15057.884703] LNet: Removed LNI 192.168.203.109@tcp19 [15057.899475] LNet: Removed LNI 192.168.203.109@tcp20 [15057.908164] LNet: Removed LNI 192.168.203.109@tcp21 [15057.912782] LNet: Removed LNI 192.168.203.109@tcp22 [15057.919718] LNet: Removed LNI 192.168.203.109@tcp23 [15057.926965] LNet: Removed LNI 192.168.203.109@tcp24 [15057.934192] LNet: Removed LNI 192.168.203.109@tcp25 [15057.939967] LNet: Removed LNI 192.168.203.109@tcp26 [15057.941984] LNet: Removed LNI 192.168.203.109@tcp27 [15057.958138] LNet: Removed LNI 192.168.203.109@tcp28 [15057.961301] LNet: Removed LNI 192.168.203.109@tcp29 [15057.964408] LNet: Removed LNI 192.168.203.109@tcp30 [15057.971220] LNet: Removed LNI 192.168.203.109@tcp31 [15057.977598] LNet: Removed LNI 192.168.203.109@tcp32 [15057.982603] LNet: Removed LNI 192.168.203.109@tcp33 [15057.985368] LNet: Removed LNI 192.168.203.109@tcp34 [15057.988576] LNet: Removed LNI 192.168.203.109@tcp35 [15057.998566] LNet: Removed LNI 192.168.203.109@tcp36 [15058.004788] LNet: Removed LNI 192.168.203.109@tcp37 [15058.010492] LNet: Removed LNI 192.168.203.109@tcp38 [15058.017371] LNet: Removed LNI 192.168.203.109@tcp39 [15058.020563] LNet: Removed LNI 192.168.203.109@tcp40 [15058.022485] LNet: Removed LNI 192.168.203.109@tcp41 [15058.025203] LNet: Removed LNI 192.168.203.109@tcp42 [15058.031115] LNet: Removed LNI 192.168.203.109@tcp43 [15058.046798] LNet: Removed LNI 192.168.203.109@tcp44 [15058.050324] LNet: Removed LNI 192.168.203.109@tcp45 [15058.053559] LNet: Removed LNI 192.168.203.109@tcp46 [15058.057175] LNet: Removed LNI 192.168.203.109@tcp47 [15058.061227] LNet: Removed LNI 192.168.203.109@tcp48 [15058.066425] LNet: Removed LNI 192.168.203.109@tcp49 [15058.082344] LNet: Removed LNI 192.168.203.109@tcp50 [15059.066312] Key type .llcrypt unregistered [15059.070773] Key type ._llcrypt unregistered [15082.424448] Key type ._llcrypt registered [15082.426767] Key type .llcrypt registered [15082.523630] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [15099.493074] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [15100.978194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15101.078432] alg: No test for adler32 (adler32-zlib) [15102.267027] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [15102.599375] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [15104.319199] Key type lgssc registered [15105.616890] Lustre: Echo OBD driver; http://www.lustre.org/ [15145.949746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15145.984242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15147.357592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15147.403089] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15147.511483] Lustre: lustre-MDT0000: new disk, initializing [15147.603455] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15147.634745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15152.568254] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [15162.700852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15162.865257] Lustre: 505428:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15162.926423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15162.932618] Lustre: Skipped 1 previous similar message [15163.018403] Lustre: lustre-MDT0001: new disk, initializing [15163.122551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15163.155719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15163.175599] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15167.819441] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [15178.808761] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15185.496823] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15194.766891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15195.012888] Lustre: *** cfs_fail_loc=90f, val=0*** [15195.113208] Lustre: lustre-OST0000: new disk, initializing [15195.116981] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15195.123072] Lustre: 507444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15195.189795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15202.354200] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [15204.401829] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15204.414692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15204.554449] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15213.022937] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15225.326891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15225.341891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15225.342345] LustreError: Skipped 1 previous similar message [15225.344186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15225.358932] Lustre: Skipped 1 previous similar message [15225.377451] Lustre: Skipped 1 previous similar message [15227.539762] Lustre: server umount lustre-OST0000 complete [15235.042135] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15235.051912] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15235.073064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15238.691403] Lustre: server umount lustre-MDT0000 complete [15243.032812] LustreError: 505420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634455 with bad export cookie 9146891343373453931 [15243.035714] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15243.062708] LustreError: 505420:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15243.471653] Lustre: server umount lustre-MDT0001 complete [15257.272166] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [15260.342748] Key type lgssc unregistered [15260.682775] LNet: 509350:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15260.689682] LNetError: 509350:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15260.708723] LNet: Removed LNI 192.168.203.109@tcp [15261.686540] Key type .llcrypt unregistered [15261.690549] Key type ._llcrypt unregistered [15284.444083] Key type ._llcrypt registered [15284.453859] Key type .llcrypt registered [15284.580331] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [15285.515281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15285.556874] alg: No test for adler32 (adler32-zlib) [15286.644939] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [15286.818603] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [15288.463138] Key type lgssc registered [15289.547310] Lustre: Echo OBD driver; http://www.lustre.org/ [15297.065589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15297.129331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15298.703991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15303.036512] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [15308.264336] LustreError: 510614:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15308.285291] LustreError: 510614:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15311.338075] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15311.957220] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15316.486731] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [15326.487781] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15333.027720] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15340.059107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15340.458847] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15345.992782] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [15353.838953] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15369.279677] loop0: detected capacity change from 0 to 204800000 [15369.680285] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15369.728372] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15369.766941] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15377.548620] loop0: detected capacity change from 0 to 204800000 [15377.792403] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15377.826812] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15377.859130] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15383.732271] loop0: detected capacity change from 0 to 204800000 [15391.102392] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15391.376347] Lustre: fs15246-OST0000: new disk, initializing [15391.381628] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15391.390892] Lustre: 514208:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15391.499417] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15398.727760] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [15409.128685] loop1: detected capacity change from 0 to 204800000 [15415.572929] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15415.653367] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15415.697608] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15415.759869] Lustre: fs15246-MDT0000: new disk, initializing [15415.879188] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15415.900207] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15415.975505] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15416.992058] Lustre: 515332:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000abce014c x1875519207425920/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788634644 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15422.005913] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [15425.197199] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15425.210538] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15425.255443] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15449.068452] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15449.083623] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15451.033690] Lustre: server umount fs15246-MDT0000 complete [15454.201387] LustreError: 512455:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15459.298076] LustreError: 510610:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15462.154951] Lustre: server umount fs15246-OST0000 complete [15474.657285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15474.658997] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15474.682610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15479.791765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15479.803585] Lustre: Skipped 2 previous similar messages [15480.400349] Lustre: server umount lustre-OST0000 complete [15484.897172] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15484.900959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15484.915180] Lustre: Skipped 2 previous similar messages [15484.935791] Lustre: Skipped 1 previous similar message [15490.021039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15490.035863] Lustre: Skipped 1 previous similar message [15499.231086] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15499.478169] Lustre: server umount lustre-MDT0000 complete [15500.261724] LustreError: 510611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15500.288966] LustreError: 510611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15503.584589] LustreError: 512459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634715 with bad export cookie 2231378442821972528 [15503.587650] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15503.597124] LustreError: 512459:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15504.192511] Lustre: server umount lustre-MDT0001 complete [15518.275208] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [15522.210831] Key type lgssc unregistered [15522.679397] LNet: 518430:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15522.689570] LNetError: 518430:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15522.704744] LNet: Removed LNI 192.168.203.109@tcp [15523.910376] Key type .llcrypt unregistered [15523.911921] Key type ._llcrypt unregistered [15543.848504] Key type ._llcrypt registered [15543.851627] Key type .llcrypt registered [15543.990222] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [15545.173392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15545.475718] alg: No test for adler32 (adler32-zlib) [15546.562872] Lustre: Lustre: Build Version: 2.17.58_3_g10061d6 [15546.811444] LNet: Added LNI 192.168.203.109@tcp [8/256/0/180] [15548.511132] Key type lgssc registered [15549.660302] Lustre: Echo OBD driver; http://www.lustre.org/ [15588.830157] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [15599.541535] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [15637.990942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15638.017174] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15639.302350] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15639.393853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15639.499782] Lustre: lustre-MDT0000: new disk, initializing [15639.599831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15639.623804] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15644.505805] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [15656.272211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15656.409399] Lustre: 522896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15656.476583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15656.480441] Lustre: Skipped 1 previous similar message [15656.572940] Lustre: lustre-MDT0001: new disk, initializing [15656.682204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15656.720172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15656.731474] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15662.103440] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [15672.721646] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15679.567435] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15687.038342] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15687.304242] Lustre: lustre-OST0000: new disk, initializing [15687.327483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15687.355282] Lustre: 524876:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15687.488283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15693.404659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15693.421786] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15693.521720] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15694.398594] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_default_debug -1 all [15704.813365] Lustre: DEBUG MARKER: oleg309-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15709.204909] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15709.452919] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15714.238574] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15714.459501] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15719.393489] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15719.408240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15719.432224] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15722.674136] Lustre: server umount lustre-OST0000 complete [15728.608407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15728.626147] LustreError: Skipped 1 previous similar message [15728.628128] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15728.645267] Lustre: Skipped 1 previous similar message [15728.655154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15728.673175] Lustre: Skipped 1 previous similar message [15729.635820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15729.655801] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15729.672812] Lustre: Skipped 1 previous similar message [15733.193240] Lustre: server umount lustre-MDT0000 complete [15737.109473] LustreError: 522885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788634949 with bad export cookie 15518285176240926296 [15737.115136] LustreError: MGC192.168.203.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15737.118349] LustreError: 522885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15737.663268] Lustre: server umount lustre-MDT0001 complete [15763.820678] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing set_hostid [15775.744649] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing load_modules_local [15811.982602] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15537 sec ========= 15:03:42 (1788635022) [15813.913436] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:03:44 (1788635024) === [15817.765234] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:03:47 (1788635027) === [15860.130428] Lustre: DEBUG MARKER: oleg309-server.virtnet: executing unload_modules_local [15862.816348] Key type lgssc unregistered [15863.146540] LNet: 531522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15863.162843] LNetError: 531522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15863.178821] LNet: Removed LNI 192.168.203.109@tcp [15864.048703] Key type .llcrypt unregistered [15864.050444] Key type ._llcrypt unregistered