[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 501543218 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002375] x2apic enabled [ 0.003007] Switched APIC routing to physical x2apic. [ 0.004027] kvm-guest: setup PV IPIs [ 0.007351] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010155] LSM: Security Framework initializing [ 0.011051] Yama: becoming mindful. [ 0.013030] SELinux: Initializing. [ 0.014000] *** VALIDATE selinux *** [ 0.021072] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025383] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026147] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029103] *** VALIDATE tmpfs *** [ 0.030692] *** VALIDATE proc *** [ 0.031387] *** VALIDATE cgroup *** [ 0.032020] *** VALIDATE cgroup2 *** [ 0.034330] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036129] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037015] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038048] Spectre V2 : User space: Vulnerable [ 0.039018] Speculative Store Bypass: Vulnerable [ 0.042632] debug: unmapping init [mem 0xffffffffa2c59000-0xffffffffa2c60fff] [ 0.045000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045742] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046031] ... version: 2 [ 0.047014] ... bit width: 48 [ 0.048017] ... generic registers: 4 [ 0.049024] ... value mask: 0000ffffffffffff [ 0.050037] ... max period: 00007fffffffffff [ 0.051023] ... fixed-purpose events: 3 [ 0.052019] ... event mask: 000000070000000f [ 0.053409] rcu: Hierarchical SRCU implementation. [ 0.055916] smp: Bringing up secondary CPUs ... [ 0.056743] x86: Booting SMP configuration: [ 0.057040] .... node #0, CPUs: #1 #2 #3 [ 0.061128] smp: Brought up 1 node, 4 CPUs [ 0.063024] smpboot: Max logical packages: 1 [ 0.064027] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.147000] node 0 deferred pages initialised in 79ms [ 0.151012] devtmpfs: initialized [ 0.152268] x86/mm: Memory block size: 128MB [ 0.155111] gcov: version magic: 0x41383552 [ 0.157550] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.161146] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.163496] pinctrl core: initialized pinctrl subsystem [ 0.166312] [ 0.166935] ************************************************************* [ 0.169022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.171017] ** ** [ 0.173019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.175022] ** ** [ 0.178017] ** This means that this kernel is built to expose internal ** [ 0.180017] ** IOMMU data structures, which may compromise security on ** [ 0.182019] ** your system. ** [ 0.184022] ** ** [ 0.187019] ** If you see this message and you are not debugging the ** [ 0.189015] ** kernel, report this immediately to your vendor! ** [ 0.191021] ** ** [ 0.193022] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.196023] ************************************************************* [ 0.198891] NET: Registered protocol family 16 [ 0.200535] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.203119] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.206101] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.210051] cpuidle: using governor menu [ 0.211874] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.212557] PCI: Using configuration type 1 for base access [ 0.213152] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.221175] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.224041] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.227176] cryptd: max_cpu_qlen set to 1000 [ 0.230233] ACPI: Added _OSI(Module Device) [ 0.232032] ACPI: Added _OSI(Processor Device) [ 0.233012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.235022] ACPI: Added _OSI(Processor Aggregator Device) [ 0.239967] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.247589] ACPI: Interpreter enabled [ 0.249097] ACPI: PM: (supports S0 S3 S4 S5) [ 0.251021] ACPI: Using IOAPIC for interrupt routing [ 0.253171] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.257451] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.270792] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.273063] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.276033] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.280128] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.286588] acpiphp: Slot [2] registered [ 0.288221] acpiphp: Slot [5] registered [ 0.289184] acpiphp: Slot [6] registered [ 0.291195] acpiphp: Slot [7] registered [ 0.293184] acpiphp: Slot [8] registered [ 0.294197] acpiphp: Slot [9] registered [ 0.296301] acpiphp: Slot [10] registered [ 0.298208] acpiphp: Slot [3] registered [ 0.299138] acpiphp: Slot [4] registered [ 0.301189] acpiphp: Slot [11] registered [ 0.303119] acpiphp: Slot [12] registered [ 0.304130] acpiphp: Slot [13] registered [ 0.306137] acpiphp: Slot [14] registered [ 0.307175] acpiphp: Slot [15] registered [ 0.309159] acpiphp: Slot [16] registered [ 0.310132] acpiphp: Slot [17] registered [ 0.312245] acpiphp: Slot [18] registered [ 0.314148] acpiphp: Slot [19] registered [ 0.315193] acpiphp: Slot [20] registered [ 0.316193] acpiphp: Slot [21] registered [ 0.318128] acpiphp: Slot [22] registered [ 0.319154] acpiphp: Slot [23] registered [ 0.321115] acpiphp: Slot [24] registered [ 0.322172] acpiphp: Slot [25] registered [ 0.324169] acpiphp: Slot [26] registered [ 0.325191] acpiphp: Slot [27] registered [ 0.327174] acpiphp: Slot [28] registered [ 0.328151] acpiphp: Slot [29] registered [ 0.330175] acpiphp: Slot [30] registered [ 0.331138] acpiphp: Slot [31] registered [ 0.333080] PCI host bridge to bus 0000:00 [ 0.334025] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.336032] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.339033] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.341030] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.343042] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.346037] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.347210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.350013] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353415] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.363021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.370063] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.373027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.375027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.377026] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.379508] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.382812] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385050] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.387853] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.393019] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.405026] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.409942] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.416000] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.422022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.430019] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.445023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.456969] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.469023] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.476025] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.499025] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.510327] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.520021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.533018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.547018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.558348] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.569024] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.578025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.604025] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.616192] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.624024] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.630022] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.645024] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.655615] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.663022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.670023] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.689024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.702708] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.705308] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.707332] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.709452] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.712263] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.717200] iommu: Default domain type: Passthrough [ 0.718569] SCSI subsystem initialized [ 0.720263] ACPI: bus type USB registered [ 0.722345] usbcore: registered new interface driver usbfs [ 0.724145] usbcore: registered new interface driver hub [ 0.726104] usbcore: registered new device driver usb [ 0.727201] pps_core: LinuxPPS API ver. 1 registered [ 0.729021] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.732139] PTP clock support registered [ 0.735066] EDAC MC: Ver: 3.0.0 [ 0.738235] PCI: Using ACPI for IRQ routing [ 0.739505] NetLabel: Initializing [ 0.740000] NetLabel: domain hash size = 128 [ 0.742026] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.744127] NetLabel: unlabeled traffic allowed by default [ 0.746161] vgaarb: loaded [ 0.748219] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.750013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.757496] clocksource: Switched to clocksource kvm-clock [ 0.871579] VFS: Disk quotas dquot_6.6.0 [ 0.873437] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.876309] *** VALIDATE ramfs *** [ 0.877756] *** VALIDATE hugetlbfs *** [ 0.879629] pnp: PnP ACPI init [ 0.882303] pnp: PnP ACPI: found 6 devices [ 0.904193] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.907423] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.909757] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.911975] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.914480] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.917071] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.919991] NET: Registered protocol family 2 [ 0.922688] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.927832] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.931465] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.937045] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.939916] TCP: Hash tables configured (established 65536 bind 65536) [ 0.943171] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.946594] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.949437] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.952272] NET: Registered protocol family 1 [ 0.956050] RPC: Registered named UNIX socket transport module. [ 0.958432] RPC: Registered udp transport module. [ 0.960242] RPC: Registered tcp transport module. [ 0.962142] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.964836] NET: Registered protocol family 44 [ 0.966838] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.969308] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.972230] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.974815] PCI: CLS 0 bytes, default 64 [ 0.976695] Unpacking initramfs... [ 2.419092] debug: unmapping init [mem 0xffff8dc37cc54000-0xffff8dc37ffbffff] [ 2.422901] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.425099] software IO TLB: mapped [mem 0x00000000a1a00000-0x00000000a5a00000] (64MB) [ 2.429968] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.946408] Initialise system trusted keyrings [ 2.948209] Key type blacklist registered [ 2.951569] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.962923] zbud: loaded [ 2.966980] *** VALIDATE nfs *** [ 2.968445] *** VALIDATE nfs4 *** [ 2.970230] pstore: using deflate compression [ 2.973766] Platform Keyring initialized [ 3.104565] NET: Registered protocol family 38 [ 3.106476] Key type asymmetric registered [ 3.108179] Asymmetric key parser 'x509' registered [ 3.109918] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.113262] io scheduler mq-deadline registered [ 3.114859] io scheduler kyber registered [ 3.116471] io scheduler bfq registered [ 3.118353] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.121491] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.124620] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.127635] ACPI: Power Button [PWRF] [ 3.133412] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.140671] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.153834] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.160405] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.174022] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.204893] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.237632] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.243258] Non-volatile memory driver v1.3 [ 3.245018] Linux agpgart interface v0.103 [ 3.283427] virtio_blk virtio1: [vda] 146008 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.288144] vda: detected capacity change from 0 to 74756096 [ 3.303103] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.306131] vdb: detected capacity change from 0 to 1073741824 [ 3.323749] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.326826] vdc: detected capacity change from 0 to 2621440000 [ 3.342483] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.345216] vdd: detected capacity change from 0 to 2621440000 [ 3.359322] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.362267] vde: detected capacity change from 0 to 4294967296 [ 3.386470] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.388477] vdf: detected capacity change from 0 to 4294967296 [ 3.401100] libphy: Fixed MDIO Bus: probed [ 3.414651] usbcore: registered new interface driver usbserial_generic [ 3.417315] usbserial: USB Serial support registered for generic [ 3.419894] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.424891] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.426483] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.428850] mousedev: PS/2 mouse device common for all mice [ 3.432225] rtc_cmos 00:05: RTC can wake from S4 [ 3.435601] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.438612] rtc_cmos 00:05: registered as rtc0 [ 3.441084] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.442324] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.444131] intel_pstate: CPU model not supported [ 3.449982] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.452734] hid: raw HID events driver (C) Jiri Kosina [ 3.456162] usbcore: registered new interface driver usbhid [ 3.457944] usbhid: USB HID core driver [ 3.459398] drop_monitor: Initializing network drop monitor service [ 3.461624] Initializing XFRM netlink socket [ 3.463455] NET: Registered protocol family 10 [ 3.467057] Segment Routing with IPv6 [ 3.468506] NET: Registered protocol family 17 [ 3.472215] mpls_gso: MPLS GSO support [ 3.479112] RAS: Correctable Errors collector initialized. [ 3.480986] AVX version of gcm_enc/dec engaged. [ 3.482180] AES CTR mode by8 optimization enabled [ 3.561822] sched_clock: Marking stable (3561800409, 0)->(4443175070, -881374661) [ 3.565559] registered taskstats version 1 [ 3.568357] Loading compiled-in X.509 certificates [ 3.569992] zswap: loaded using pool lzo/zbud [ 3.598182] Key type big_key registered [ 3.613618] Key type encrypted registered [ 3.615760] ima: No TPM chip found, activating TPM-bypass! [ 3.617919] ima: Allocated hash algorithm: sha1 [ 3.619871] ima: No architecture policies found [ 3.622150] evm: Initialising EVM extended attributes: [ 3.624169] evm: security.selinux [ 3.625584] evm: security.ima [ 3.626722] evm: security.capability [ 3.628349] evm: HMAC attrs: 0x1 [ 3.631224] rtc_cmos 00:05: setting system clock to 2026-08-22 01:01:05 UTC (1787360465) [ 3.638246] debug: unmapping init [mem 0xffffffffa3c03000-0xffffffffa3dfffff] [ 3.641786] debug: unmapping init [mem 0xffffffffa2982000-0xffffffffa2c58fff] [ 3.649472] Write protecting the kernel read-only data: 28672k [ 3.653619] debug: unmapping init [mem 0xffffffffa1003000-0xffffffffa11fffff] [ 3.656893] debug: unmapping init [mem 0xffffffffa1914000-0xffffffffa19fffff] [ 3.693656] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.701932] systemd[1]: Detected virtualization kvm. [ 3.703977] systemd[1]: Detected architecture x86-64. [ 3.705823] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.736675] systemd[1]: No hostname configured. [ 3.738071] systemd[1]: Set hostname to . [ 3.739882] random: systemd: uninitialized urandom read (16 bytes read) [ 3.741954] systemd[1]: Initializing machine ID from random generator. [ 3.789325] random: ln: uninitialized urandom read (6 bytes read) [ 3.888703] random: systemd: uninitialized urandom read (16 bytes read) [ 3.890933] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.896193] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ 3.901402] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... Starting Journal Service... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.625801] device-mapper: uevent: version 1.0.3 [ 4.628200] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.455439] virtio_net virtio0 ens2: renamed from eth0 [ 5.485192] random: fast init done [ 5.543217] scsi host0: ata_piix [ 5.564863] scsi host1: ata_piix [ 5.566639] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.569134] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.368589] dracut-initqueue[594]: RTNETLINK answers: File exists [ 10.135554] random: crng init done [ 10.136756] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.731610] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.976509] printk: systemd: 25 output lines suppressed due to ratelimiting [ 12.289167] SELinux: Disabled at runtime. [ 12.350886] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.359510] systemd[1]: Detected virtualization kvm. [ 12.361365] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.907199] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.910943] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.916451] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.920195] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.923698] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.931972] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.936421] systemd[1]: Stopped target Switch Root. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Kernel Debug File System... [ 13.024338] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Paths. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Reached target rpc_pipefs.target. [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Configure read-only root support... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 13.417883] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.817829] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.840237] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.070598] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.085253] EDAC sbridge: Ver: 1.1.2 [ 15.924945] Key type dns_resolver registered [ 16.244076] NFS: Registering the id_resolver key type [ 16.246062] Key type id_resolver registered [ 16.247915] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... Starting Login Service... Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg426-server login: [ 43.087074] libcfs: loading out-of-tree module taints kernel. [ 43.114446] Key type ._llcrypt registered [ 43.116407] Key type .llcrypt registered [ 43.179637] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 61.612946] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 63.578252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 63.600614] alg: No test for adler32 (adler32-zlib) [ 65.430733] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 66.323442] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 68.105279] Key type lgssc registered [ 70.816539] Lustre: Echo OBD driver; http://www.lustre.org/ [ 91.984058] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 135.457149] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 150.363237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 150.435675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 151.695428] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 151.739865] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 151.840705] Lustre: lustre-MDT0000: new disk, initializing [ 151.957116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 151.980902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 156.773379] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 170.846938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 170.991814] Lustre: 6524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 171.025947] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 171.029850] Lustre: Skipped 1 previous similar message [ 171.093912] Lustre: lustre-MDT0001: new disk, initializing [ 171.164738] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 171.208627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 171.223875] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 176.429793] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 181.233351] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 190.968488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.501466] Lustre: lustre-OST0000: new disk, initializing [ 191.509456] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 191.528465] Lustre: 8462:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 191.673363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 192.688695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 192.701680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 192.814318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 198.916278] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.568569] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 215.702737] Lustre: lustre-OST0001: new disk, initializing [ 215.709149] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 215.719960] Lustre: 9535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 215.793962] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 222.189230] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 223.782581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 223.796429] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 223.866243] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 233.844113] hrtimer: interrupt took 12123728 ns [ 235.785111] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 242.999586] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 250.423411] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing check_logdir /tmp/testlogs/ [ 256.298032] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing yml_node [ 261.089731] Lustre: DEBUG MARKER: Client: 2.17.57.45 [ 264.094954] Lustre: DEBUG MARKER: MDS: 2.17.57.45 [ 266.694916] Lustre: DEBUG MARKER: OSS: 2.17.57.45 [ 268.379848] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Fri Aug 21 21:05:27 EDT 2026 [ 285.523644] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 287.170461] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 304.608345] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 304.612649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 304.621641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.632341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 305.642101] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 308.422499] Lustre: server umount lustre-MDT0000 complete [ 311.695308] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787360773 with bad export cookie 1496804787365874311 [ 311.699477] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 311.716292] LustreError: 6515:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 315.873257] LustreError: 6530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 315.874992] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.880290] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 315.880299] Lustre: Skipped 3 previous similar messages [ 315.901088] LustreError: 6530:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 315.916286] Lustre: Skipped 3 previous similar messages [ 316.896788] LustreError: 7907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 316.912081] LustreError: 7907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 322.023307] LustreError: 7907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.030021] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.057253] Lustre: Skipped 4 previous similar messages [ 326.111213] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 326.407503] Lustre: server umount lustre-MDT0001 complete [ 330.503162] Lustre: server umount lustre-OST0000 complete [ 334.788566] Lustre: server umount lustre-OST0001 complete [ 341.883405] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 349.786542] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 388.294446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 388.735183] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 388.788773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 388.913297] Lustre: lustre-MDT0000: new disk, initializing [ 389.031833] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 389.063601] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 393.714769] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 404.733656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.889976] Lustre: 15085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 404.955276] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 404.964936] Lustre: Skipped 1 previous similar message [ 405.046377] Lustre: lustre-MDT0001: new disk, initializing [ 405.147587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 405.196158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 405.207484] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 409.822692] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 420.285663] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 426.196556] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 433.304801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 433.877669] Lustre: lustre-OST0000: new disk, initializing [ 433.883295] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 433.891639] Lustre: 17054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 434.045179] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.013424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 435.028449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 435.200810] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 442.257279] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 454.342952] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 461.382727] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 461.777322] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 467.499603] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 467.897185] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 471.018690] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 471.021845] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 471.055980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 471.062509] LustreError: Skipped 1 previous similar message [ 471.084363] Lustre: Skipped 1 previous similar message [ 481.256980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 481.269821] Lustre: Skipped 3 previous similar messages [ 484.319305] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 484.561739] Lustre: server umount lustre-OST0000 complete [ 491.491828] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 491.506520] Lustre: Skipped 2 previous similar messages [ 501.735504] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 501.747694] Lustre: Skipped 5 previous similar messages [ 503.263379] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 503.480450] Lustre: server umount lustre-MDT0000 complete [ 506.848505] LustreError: 15099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 506.883663] LustreError: 15099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 507.317354] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787360969 with bad export cookie 1496804787365875844 [ 507.320598] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 507.327498] LustreError: 15080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 507.818027] Lustre: server umount lustre-MDT0001 complete [ 512.222151] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 21:09:31 (1787360971) [ 514.543676] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 516.410133] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 21:09:35 (1787360975) [ 523.290661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 523.963328] LustreError: 19469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 524.094134] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 528.514858] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 533.472796] LustreError: 19469:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 533.491755] LustreError: 19469:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 536.885614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 541.905824] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 549.490845] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 555.130192] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 560.973894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 567.187272] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 573.615292] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 583.136480] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 583.142133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 583.151732] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 586.542852] Lustre: server umount lustre-OST0000 complete [ 593.376483] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 593.390163] LustreError: Skipped 1 previous similar message [ 596.465598] Lustre: server umount lustre-MDT0000 complete [ 599.526314] LustreError: 19470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 599.544218] LustreError: 19470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 600.184576] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787361062 with bad export cookie 1496804787365876726 [ 600.190784] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 600.196540] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 600.496433] Lustre: server umount lustre-MDT0001 complete [ 611.440720] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 614.286842] Key type lgssc unregistered [ 614.613531] LNet: 23004:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 614.626659] LNetError: 23004:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 614.640894] LNet: Removed LNI 192.168.204.126@tcp [ 615.713753] Key type .llcrypt unregistered [ 615.716845] Key type ._llcrypt unregistered [ 638.570042] Key type ._llcrypt registered [ 638.574352] Key type .llcrypt registered [ 638.699403] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 639.825863] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 639.884821] alg: No test for adler32 (adler32-zlib) [ 641.036281] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 641.314604] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 643.024164] Key type lgssc registered [ 644.074554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 654.170389] loop: module loaded [ 654.265694] loop0: detected capacity change from 0 to 204800000 [ 654.460041] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 654.477967] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 654.830655] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 661.206386] loop0: detected capacity change from 0 to 204800000 [ 661.394337] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 661.401393] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 661.410482] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 666.793780] loop0: detected capacity change from 0 to 204800000 [ 673.333627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 673.355320] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 674.649901] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 674.733992] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 674.844703] Lustre: lustre-MDT0000: new disk, initializing [ 674.921842] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 674.965523] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 680.122232] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 688.501052] loop1: detected capacity change from 0 to 204800000 [ 694.058543] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 694.313229] Lustre: lustre-OST07c6: new disk, initializing [ 694.317670] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 694.322341] Lustre: Skipped 1 previous similar message [ 694.332525] Lustre: 26195:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 694.344848] Lustre: lustre-OST07c6: Not available for connect from 0@lo (not set up) [ 694.388649] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 699.732555] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 699.945192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 699.958356] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 699.989145] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 704.950729] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 704.959465] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 704.966236] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 704.972564] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 704.980332] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 704.988370] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 717.475388] Lustre: 25177:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 717.484436] Lustre: 25177:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 717.490784] Lustre: 25177:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 717.505304] Lustre: 25177:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 717.511437] Lustre: 25177:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 717.515982] Lustre: 25177:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.513400] Lustre: 25667:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 722.521130] Lustre: 25667:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 722.526037] Lustre: 25667:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 722.530921] Lustre: 25667:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.536431] Lustre: 25667:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 722.541410] Lustre: 25667:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.550933] Lustre: 25667:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 722.556036] Lustre: 25667:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.561271] Lustre: 25667:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 722.567449] Lustre: 25667:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 722.572972] Lustre: 25667:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 722.578603] Lustre: 25667:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.746272] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 725.758671] Lustre: 25178:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 725.770662] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 725.794588] Lustre: 25178:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.812593] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 725.827638] Lustre: 25178:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.839633] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 725.853423] Lustre: 25178:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.860146] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 725.864801] Lustre: 25178:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 725.870139] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 725.874490] Lustre: 25178:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.663296] Lustre: 25667:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 730.677201] Lustre: 25667:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 730.688254] Lustre: 25667:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 730.697035] Lustre: 25667:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.702331] Lustre: 25667:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 730.707114] Lustre: 25667:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.712862] Lustre: 25667:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 730.718953] Lustre: 25667:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.730947] Lustre: 25667:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 730.738050] Lustre: 25667:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 730.747051] Lustre: 25667:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 730.758939] Lustre: 25667:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 742.246609] Lustre: 25667:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 742.255975] Lustre: 25667:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 742.262047] Lustre: 25667:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 742.269450] Lustre: 25667:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 742.274807] Lustre: 25667:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 742.283352] Lustre: 25667:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 742.291894] Lustre: 25667:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 742.302540] Lustre: 25667:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 742.308806] Lustre: 25667:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 742.316176] Lustre: 25667:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 742.322940] Lustre: 25667:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 742.330576] Lustre: 25667:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.042044] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 792.073325] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 792.096562] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 796.018800] Lustre: server umount lustre-OST07c6 complete [ 810.112127] Lustre: server umount lustre-MDT0000 complete [ 827.964708] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 831.336773] Key type lgssc unregistered [ 831.633170] LNet: 28755:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 831.646475] LNetError: 28755:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 831.663068] LNet: Removed LNI 192.168.204.126@tcp [ 832.549591] Key type .llcrypt unregistered [ 832.552676] Key type ._llcrypt unregistered [ 853.406119] Key type ._llcrypt registered [ 853.409063] Key type .llcrypt registered [ 853.621445] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 855.358313] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 855.421059] alg: No test for adler32 (adler32-zlib) [ 856.610728] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 856.916394] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 858.658129] Key type lgssc registered [ 860.258911] Lustre: Echo OBD driver; http://www.lustre.org/ [ 868.959457] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 868.984295] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 870.652926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 876.440423] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 879.011913] LustreError: 30028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 879.050317] LustreError: 30028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 881.062244] LustreError: 30029:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 886.246949] LustreError: 30028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 888.096807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 888.549714] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 894.017163] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 902.208700] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 909.292959] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 917.841218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 918.329482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 928.129770] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 936.306335] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 940.870568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 959.457949] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 959.467634] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 959.484220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 961.517984] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 961.521116] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 961.542071] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 961.549032] Lustre: Skipped 1 previous similar message [ 963.465750] Lustre: server umount lustre-OST0000 complete [ 970.208966] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 970.219609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 970.245220] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 974.439838] Lustre: server umount lustre-MDT0000 complete [ 976.867174] LustreError: 30023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 976.889702] LustreError: 30023:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 978.070860] LustreError: 32836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787361439 with bad export cookie 13279992552851853633 [ 978.078626] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 978.084844] LustreError: 32836:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 978.748056] Lustre: server umount lustre-MDT0001 complete [ 995.615251] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 998.792357] Key type lgssc unregistered [ 999.163905] LNet: 33795:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 999.174770] LNetError: 33795:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 999.199421] LNet: Removed LNI 192.168.204.126@tcp [ 1000.394982] Key type .llcrypt unregistered [ 1000.398797] Key type ._llcrypt unregistered [ 1020.452733] Key type ._llcrypt registered [ 1020.454940] Key type .llcrypt registered [ 1020.548257] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1021.819136] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1021.976768] alg: No test for adler32 (adler32-zlib) [ 1023.104592] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1023.391451] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1025.114056] Key type lgssc registered [ 1026.280418] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1033.633995] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1033.670856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1035.288127] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1039.636704] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1045.478107] LustreError: 35066:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1045.509061] LustreError: 35066:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1047.756622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1048.123744] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1052.419030] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1058.954129] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1063.894413] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1069.572672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1070.029116] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.948434] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1080.316841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1086.298958] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1098.725724] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1098.737181] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1098.757472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1100.773220] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1100.774304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1100.793304] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.021883] Lustre: server umount lustre-OST0000 complete [ 1109.472594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1109.477149] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1109.496340] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1109.501360] Lustre: Skipped 1 previous similar message [ 1113.579525] Lustre: server umount lustre-MDT0000 complete [ 1116.136119] LustreError: 36088:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1116.171206] LustreError: 36088:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1117.820315] LustreError: 35048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787361579 with bad export cookie 12860341894518537273 [ 1117.823500] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1117.833831] LustreError: 35048:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1118.340827] Lustre: server umount lustre-MDT0001 complete [ 1130.016642] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1133.255629] Key type lgssc unregistered [ 1133.651191] LNet: 38603:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1133.657329] LNetError: 38603:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1134.693830] LNet: Removed LNI 192.168.204.126@tcp [ 1135.542612] Key type .llcrypt unregistered [ 1135.544528] Key type ._llcrypt unregistered [ 1160.037268] Key type ._llcrypt registered [ 1160.042753] Key type .llcrypt registered [ 1160.179738] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1161.188924] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1161.209691] alg: No test for adler32 (adler32-zlib) [ 1162.268620] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1162.456539] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1164.192097] Key type lgssc registered [ 1165.765768] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1176.660185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1176.687034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1178.194340] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1184.178348] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1187.297237] LustreError: 40022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1187.309784] LustreError: 40022:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1192.422976] LustreError: 40023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1193.773050] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1194.333650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1199.696070] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1207.163348] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1212.716112] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1220.329127] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1220.865973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1222.914806] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1227.451740] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1234.500395] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1245.665091] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1245.674964] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1245.712393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1247.296560] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1249.765221] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1249.766969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1249.789198] Lustre: Skipped 1 previous similar message [ 1251.098924] Lustre: server umount lustre-MDT0000 complete [ 1252.384305] LustreError: 40018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.888593] LustreError: 41578:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1254.908841] LustreError: 41578:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1255.130936] LustreError: 40004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787361716 with bad export cookie 8528016314896317176 [ 1255.147394] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1257.539107] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 1260.000799] LustreError: 42465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1260.003949] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1260.017263] LustreError: 42465:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1260.031132] Lustre: Skipped 1 previous similar message [ 1261.676845] Lustre: server umount lustre-MDT0001 complete [ 1276.842805] Lustre: server umount lustre-OST0000 complete [ 1292.254611] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1295.165328] Key type lgssc unregistered [ 1295.429760] LNet: 43885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1295.437909] LNetError: 43885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1295.452201] LNet: Removed LNI 192.168.204.126@tcp [ 1296.122275] Key type .llcrypt unregistered [ 1296.124813] Key type ._llcrypt unregistered [ 1319.167985] Key type ._llcrypt registered [ 1319.171395] Key type .llcrypt registered [ 1319.293876] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1320.170467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1320.197688] alg: No test for adler32 (adler32-zlib) [ 1321.258568] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1321.461799] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1323.119444] Key type lgssc registered [ 1324.329341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1332.706834] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1332.723006] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1334.164872] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1339.240944] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1343.462961] LustreError: 45301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1343.479928] LustreError: 45301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1348.090321] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.488523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1353.231357] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1360.629778] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1366.880182] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1375.405283] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1376.011656] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1379.127204] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1383.190535] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1393.854987] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1401.917951] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1403.872953] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1403.877064] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1403.886429] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1403.895516] Lustre: Skipped 1 previous similar message [ 1407.003074] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1407.011698] Lustre: Skipped 1 previous similar message [ 1407.739405] Lustre: server umount lustre-OST0000 complete [ 1428.450680] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1428.474694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1428.480503] Lustre: Skipped 1 previous similar message [ 1428.495542] Lustre: Skipped 1 previous similar message [ 1433.575954] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1433.581424] Lustre: Skipped 1 previous similar message [ 1440.223477] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1440.515438] Lustre: server umount lustre-MDT0000 complete [ 1443.815162] LustreError: 45301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1443.841534] LustreError: 45301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1444.408483] LustreError: 45282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787361906 with bad export cookie 13039896593600087998 [ 1444.423903] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1450.804818] Lustre: server umount lustre-MDT0001 complete [ 1463.617190] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1466.897165] Key type lgssc unregistered [ 1467.215166] LNet: 49017:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1467.228984] LNetError: 49017:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1467.246343] LNet: Removed LNI 192.168.204.126@tcp [ 1468.427048] Key type .llcrypt unregistered [ 1468.430644] Key type ._llcrypt unregistered [ 1494.906951] Key type ._llcrypt registered [ 1494.912195] Key type .llcrypt registered [ 1495.031530] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1495.819949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1495.859616] alg: No test for adler32 (adler32-zlib) [ 1496.887587] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1497.096639] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1498.799748] Key type lgssc registered [ 1499.829956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1507.158462] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1507.176449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1508.521571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1512.462963] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1519.076831] LustreError: 50433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1519.117758] LustreError: 50433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1524.228705] LustreError: 50432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1524.323863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1524.987180] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1530.536565] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1539.538975] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1546.111865] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1553.224782] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1553.807610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1554.866942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1560.646506] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1570.397254] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1576.713473] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1581.439326] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1594.573630] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2121h27m32s [ 1596.309875] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1599.988884] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1600.017876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1600.023881] Lustre: Skipped 1 previous similar message [ 1600.055267] Lustre: Skipped 2 previous similar messages [ 1601.159709] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1601.184266] Lustre: Skipped 2 previous similar messages [ 1603.146953] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 1605.010412] Lustre: server umount lustre-MDT0000 complete [ 1607.654567] LustreError: 50432:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1607.692236] LustreError: 50432:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1609.789544] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1610.266841] LustreError: 50429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1616.424238] LustreError: 51584:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1616.444117] LustreError: 51584:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1616.490239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1616.627402] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1616.929439] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1617.035436] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1617.973952] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1621.966874] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1625.628530] Lustre: DEBUG MARKER: Wait for df (36700) ... [ 1627.289616] Lustre: DEBUG MARKER: done [ 1632.225066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1632.232787] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.255080] Lustre: Skipped 1 previous similar message [ 1632.261937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.268473] Lustre: Skipped 1 previous similar message [ 1633.262452] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1638.376456] LustreError: 52255:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1638.408316] LustreError: 52255:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1638.633641] Lustre: server umount lustre-OST0000 complete [ 1643.494070] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1643.497542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1643.510901] Lustre: Skipped 1 previous similar message [ 1643.520190] Lustre: Skipped 3 previous similar messages [ 1649.506406] Lustre: server umount lustre-MDT0000 complete [ 1654.014562] LustreError: 50415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787362115 with bad export cookie 17772805130302068079 [ 1654.018102] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1654.027599] LustreError: 50415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1654.381996] Lustre: server umount lustre-MDT0001 complete [ 1666.014226] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1669.132551] Key type lgssc unregistered [ 1669.444251] LNet: 55369:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1669.450701] LNetError: 55369:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1670.506150] LNet: Removed LNI 192.168.204.126@tcp [ 1671.657510] Key type .llcrypt unregistered [ 1671.660154] Key type ._llcrypt unregistered [ 1711.331870] Key type ._llcrypt registered [ 1711.337781] Key type .llcrypt registered [ 1711.474066] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1712.561295] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1712.587793] alg: No test for adler32 (adler32-zlib) [ 1713.722227] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1713.971907] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1715.639372] Key type lgssc registered [ 1716.893495] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1724.474988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1724.502378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1725.781719] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1725.817783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1726.003270] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1730.990825] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1739.623258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.827306] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1739.860513] Lustre: 57577:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1740.180831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1745.289853] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1753.065529] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1759.272853] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1767.464275] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1767.691783] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1767.844698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1777.405700] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1777.429859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1786.519248] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1793.617541] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2121h30m52s [ 1795.504295] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1802.791291] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1805.711442] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1805.782881] LustreError: 57584:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1825.839163] Lustre: lustre-MDT0000: Client 6f883a8b-0227-4bb6-a98c-89de5d598028 (at 192.168.204.26@tcp) reconnecting [ 1845.864414] LustreError: 57584:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1845.944982] Lustre: lustre-MDT0000: Client 6f883a8b-0227-4bb6-a98c-89de5d598028 (at 192.168.204.26@tcp) reconnecting [ 1848.465105] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1851.233666] Lustre: DEBUG MARKER: done [ 1859.558661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1859.562939] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1859.598262] Lustre: Skipped 1 previous similar message [ 1859.695459] Lustre: server umount lustre-OST0000 complete [ 1864.684268] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1864.703082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1865.762535] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1865.770772] Lustre: Skipped 2 previous similar messages [ 1869.823644] Lustre: server umount lustre-MDT0000 complete [ 1870.832107] LustreError: 58586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1870.851508] LustreError: 58586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1873.427933] LustreError: 57571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787362335 with bad export cookie 2877572654485185564 [ 1873.429940] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1873.438434] LustreError: 57571:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1873.754409] Lustre: server umount lustre-MDT0001 complete [ 1885.179770] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 1888.255272] Key type lgssc unregistered [ 1888.581333] LNet: 61908:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1888.594490] LNetError: 61908:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1889.638471] LNet: Removed LNI 192.168.204.126@tcp [ 1890.604817] Key type .llcrypt unregistered [ 1890.608094] Key type ._llcrypt unregistered [ 1937.641275] Key type ._llcrypt registered [ 1937.643327] Key type .llcrypt registered [ 1937.742277] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 1938.944987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1939.099097] alg: No test for adler32 (adler32-zlib) [ 1940.226989] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 1940.480786] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 1942.159191] Key type lgssc registered [ 1943.707919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1947.727852] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1947.759096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1949.079673] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1949.104434] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1949.313257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1950.438853] Lustre: Failing over lustre-MDT0000 [ 1950.743659] Lustre: server umount lustre-MDT0000 complete [ 1958.424811] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 21:33:37 (1787362417) [ 1965.416083] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1965.920542] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1970.070799] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1979.480399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1979.604175] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1979.650985] Lustre: 64714:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1979.930284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1984.360604] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 1991.648606] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1998.381697] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2006.157988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2006.418394] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2006.697550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2011.930439] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2014.401930] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2022.114600] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2028.045993] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2036.200114] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2036.214467] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2036.248733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.732233] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.734928] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2041.659575] Lustre: server umount lustre-MDT0000 complete [ 2045.735778] LustreError: 64706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787362507 with bad export cookie 2476717848930158287 [ 2045.745575] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2045.750031] LustreError: 64706:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2046.263726] Lustre: server umount lustre-MDT0001 complete [ 2049.770478] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2055.271412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2063.271736] Lustre: 63633:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787362509/real 1787362509] req@00000000ff0bc22f x1874185318081280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787362525 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2063.325559] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.340529] Lustre: Skipped 1 previous similar message [ 2064.823394] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2066.399218] Lustre: 63635:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787362512/real 1787362512] req@00000000440b4e82 x1874185318081408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787362528 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2068.511318] Lustre: 63635:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787362514/real 1787362514] req@000000009be853f3 x1874185318081536/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787362530 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2070.982247] LustreError: 66561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2071.000939] LustreError: 66561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2071.089962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2071.163167] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2072.032702] Lustre: 63633:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787362517/real 1787362517] req@000000008b67b944 x1874185318081792/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787362533 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2072.100767] LustreError: 66562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2072.117789] LustreError: 66562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2074.148328] LustreError: 68484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2074.178339] LustreError: 68484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2075.665768] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2077.215461] Lustre: 63632:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787362523/real 1787362523] req@0000000007f27ed3 x1874185318081920/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787362539 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2079.200678] LustreError: 66562:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2079.220817] LustreError: 66562:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2084.323747] LustreError: 68481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2084.361434] LustreError: 68481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2085.842922] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.255293] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2091.499047] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2092.069212] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2092.530266] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2100.664217] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2106.465094] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2112.998108] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2113.005465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2113.020121] Lustre: Skipped 1 previous similar message [ 2113.034179] Lustre: Skipped 4 previous similar messages [ 2118.113916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2118.122352] Lustre: Skipped 2 previous similar messages [ 2123.233307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2123.249201] Lustre: Skipped 2 previous similar messages [ 2127.327186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2127.588724] Lustre: server umount lustre-MDT0000 complete [ 2128.354884] LustreError: 68481:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2128.384118] LustreError: 68481:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2131.890279] LustreError: 67624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787362593 with bad export cookie 2476717848930160562 [ 2131.891642] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2131.920858] LustreError: 67624:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2132.222723] Lustre: server umount lustre-MDT0001 complete [ 2139.786162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2149.855129] Lustre: 63635:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787362595/real 1787362595] req@0000000052a05e98 x1874185318112000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787362611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2149.893869] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2149.902354] Lustre: Skipped 1 previous similar message [ 2150.622249] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2157.090063] LustreError: 63631:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a2acf7ff x1874185318112896/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2157.838227] LustreError: 69218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2158.025749] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2158.111511] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2159.045130] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2164.682351] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2173.609497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2178.457397] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2179.061673] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2185.802042] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2191.329829] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2198.009106] Lustre: server umount lustre-MDT0000 complete [ 2199.523354] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.525672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2199.537976] Lustre: Skipped 1 previous similar message [ 2199.540314] LustreError: 69218:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2199.571407] LustreError: 69218:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 2201.902250] LustreError: 64704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787362663 with bad export cookie 2476717848930161941 [ 2201.915659] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2201.921128] LustreError: 64704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2204.243063] Lustre: server umount lustre-MDT0001 complete [ 2211.419136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2215.721746] Lustre: DEBUG MARKER: files compared the same [ 2228.405236] Lustre: server umount lustre-OST0000 complete [ 2245.210770] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2248.258312] Key type lgssc unregistered [ 2248.582856] LNet: 74450:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2248.590533] LNetError: 74450:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2248.621112] LNet: Removed LNI 192.168.204.126@tcp [ 2249.712261] Key type .llcrypt unregistered [ 2249.719109] Key type ._llcrypt unregistered [ 2274.860177] Key type ._llcrypt registered [ 2274.863706] Key type .llcrypt registered [ 2274.956355] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2276.327331] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2276.383548] alg: No test for adler32 (adler32-zlib) [ 2277.394860] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2277.630251] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2279.279719] Key type lgssc registered [ 2280.356869] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2287.497216] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2287.521621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2288.934298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2293.249456] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 2298.348141] LustreError: 75865:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2298.366485] LustreError: 75865:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2301.996484] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2302.442111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2306.769981] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 2314.652406] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2320.113452] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2326.427554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2326.766795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2332.894515] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug +malloc all [ 2333.926946] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2340.004539] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2349.545141] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2349.545692] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2349.554708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2349.572535] Lustre: Skipped 1 previous similar message [ 2354.661109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2354.665848] Lustre: Skipped 1 previous similar message [ 2354.899630] Lustre: server umount lustre-OST0000 complete [ 2358.752214] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2358.761827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2358.774489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2358.779866] Lustre: Skipped 1 previous similar message [ 2359.779765] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.904740] LustreError: 75866:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2365.089839] Lustre: server umount lustre-MDT0000 complete [ 2368.792758] LustreError: 75847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787362830 with bad export cookie 15339131874715504502 [ 2368.796245] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2368.802324] LustreError: 75847:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2369.121902] Lustre: server umount lustre-MDT0001 complete [ 2383.991100] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2386.925341] Key type lgssc unregistered [ 2387.235505] LNet: 79400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.244574] LNetError: 79400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.267484] LNet: Removed LNI 192.168.204.126@tcp [ 2388.088990] Key type .llcrypt unregistered [ 2388.091706] Key type ._llcrypt unregistered [ 2411.392829] Key type ._llcrypt registered [ 2411.396397] Key type .llcrypt registered [ 2411.533095] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2412.445700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2412.517519] alg: No test for adler32 (adler32-zlib) [ 2413.533283] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2413.806307] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2415.487853] Key type lgssc registered [ 2416.965665] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2424.961516] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2424.973467] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2440.479445] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2445.600484] LustreError: 80770:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [ 2472.287299] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2479.900751] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2488.386836] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2497.151856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2497.376563] Lustre: *** cfs_fail_loc=706, val=0*** [ 2497.776390] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2497.867604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2502.301768] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2503.136960] LustreError: 80795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2503.164721] LustreError: 80795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2506.214226] LustreError: 81856:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2506.246364] LustreError: 81856:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2510.528162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2510.955294] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2515.431385] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2523.689821] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2530.250926] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2533.857943] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2533.868917] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2533.894186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2536.417106] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2536.429116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2536.447930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2538.483851] Lustre: server umount lustre-OST0000 complete [ 2546.668180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2546.679368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2546.711643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2546.722609] Lustre: Skipped 1 previous similar message [ 2549.159493] Lustre: server umount lustre-MDT0000 complete [ 2552.289130] LustreError: 81862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2552.310311] LustreError: 81862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2553.511714] LustreError: 80776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787363015 with bad export cookie 3528597516208244697 [ 2553.512855] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2553.526089] LustreError: 80776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2554.037236] Lustre: server umount lustre-MDT0001 complete [ 2565.069337] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2567.568106] Key type lgssc unregistered [ 2567.916622] LNet: 84390:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2567.931158] LNetError: 84390:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2567.960277] LNet: Removed LNI 192.168.204.126@tcp [ 2568.879494] Key type .llcrypt unregistered [ 2568.881691] Key type ._llcrypt unregistered [ 2592.849663] Key type ._llcrypt registered [ 2592.852521] Key type .llcrypt registered [ 2592.981182] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2593.984588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2594.013486] alg: No test for adler32 (adler32-zlib) [ 2595.096567] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2595.304983] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2596.944224] Key type lgssc registered [ 2598.041533] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2605.486466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2605.514578] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2611.581623] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2621.025561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2621.918605] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2626.999667] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2627.046990] LustreError: 86494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.083102] LustreError: 86494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2631.140661] LustreError: 86498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2631.161654] LustreError: 86498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2635.700540] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2636.059977] LustreError: 86494:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2636.087386] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2636.087602] LustreError: 86494:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2641.377318] LustreError: 86497:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2641.393671] LustreError: 86497:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2642.534926] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2646.498246] LustreError: 86498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2646.524167] LustreError: 86498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2651.240983] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2651.250470] Lustre: MGS: Client 1a4e4737-8510-4e67-9d9d-421a51486309 (at 0@lo) reconnecting [ 2651.273237] LustreError: 86508:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2651.434779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2651.525891] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2655.570540] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2672.107658] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2672.108444] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2672.127813] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2672.139427] Lustre: Skipped 1 previous similar message [ 2675.720812] Lustre: server umount lustre-OST0000 complete [ 2682.338740] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2682.338869] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2682.352721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2682.352729] Lustre: Skipped 1 previous similar message [ 2682.406700] Lustre: Skipped 1 previous similar message [ 2686.356824] Lustre: server umount lustre-MDT0000 complete [ 2687.462485] LustreError: 86498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2687.485857] LustreError: 86498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2692.153525] Lustre: server umount lustre-MDT0001 complete [ 2696.977996] Lustre: server umount MGS complete [ 2712.267796] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 21:46:10 (1787363170) [ 2745.284910] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 2758.272424] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2802.423027] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2807.454189] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2818.033043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2818.336763] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2818.384515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2818.448686] Lustre: lustre-MDT0001: new disk, initializing [ 2818.513678] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2818.548608] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2818.559233] Lustre: 93682:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2818.565297] LustreError: 93682:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2823.507494] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2835.315954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2835.611107] Lustre: lustre-OST0000: new disk, initializing [ 2835.621067] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2835.629296] Lustre: 94450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2835.702780] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2843.444821] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2845.857549] Lustre: 94861:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000044b15f3f x1874186004897152/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787363323 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2854.322870] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2862.119565] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2862.140643] Lustre: MGS: Client 8e5883e5-8509-4a73-a36c-46bfc2dca2be (at 0@lo) reconnecting [ 2862.261888] Lustre: 92951:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2862.287527] LustreError: 93668:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2862.344575] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2862.439682] Lustre: lustre-MDT0000: new disk, initializing [ 2862.605038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2862.634081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2862.650993] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2863.720155] Lustre: 95519:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000276ea2f x1874186004903936/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787363341 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2867.777774] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 2868.843178] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2872.874543] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2872.891179] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2872.906083] Lustre: Skipped 1 previous similar message [ 2872.946391] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2898.401925] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2898.408668] Lustre: Skipped 1 previous similar message [ 2898.419108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2898.426337] Lustre: Skipped 2 previous similar messages [ 2903.527589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2904.407627] Lustre: server umount lustre-OST0000 complete [ 2908.640522] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2908.647272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.657556] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2908.667442] Lustre: Skipped 1 previous similar message [ 2908.689947] Lustre: Skipped 2 previous similar messages [ 2914.667961] Lustre: server umount lustre-MDT0000 complete [ 2918.882412] LustreError: 93653:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2918.921432] LustreError: 93653:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2925.732023] Lustre: server umount lustre-MDT0001 complete [ 2929.884292] Lustre: server umount MGS complete [ 2942.660929] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 21:50:01 (1787363401) [ 2958.983876] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 2961.588959] Key type lgssc unregistered [ 2961.909771] LNet: 98022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2961.921765] LNetError: 98022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2961.938295] LNet: Removed LNI 192.168.204.126@tcp [ 2962.864637] Key type .llcrypt unregistered [ 2962.868746] Key type ._llcrypt unregistered [ 2979.144430] Key type ._llcrypt registered [ 2979.149600] Key type .llcrypt registered [ 2979.289774] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 2980.695263] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2980.732363] alg: No test for adler32 (adler32-zlib) [ 2981.727469] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 2981.951693] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 2983.671415] Key type lgssc registered [ 2984.783668] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2991.711563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2991.745676] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.231620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2997.637817] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3003.373395] LustreError: 99245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3003.412481] LustreError: 99245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3007.818706] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3008.233224] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3012.857973] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3021.430687] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3026.437746] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3032.396912] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3032.717794] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3038.387088] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3038.906877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3045.812090] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3054.571136] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3054.571668] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3054.603266] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3054.616479] Lustre: Skipped 1 previous similar message [ 3059.402451] Lustre: server umount lustre-OST0000 complete [ 3064.802036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.803714] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3064.809616] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3064.826542] Lustre: Skipped 2 previous similar messages [ 3069.341837] Lustre: server umount lustre-MDT0000 complete [ 3069.921775] LustreError: 99242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3069.938490] LustreError: 99242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3072.714255] LustreError: 99225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787363534 with bad export cookie 10626716746934063342 [ 3072.714720] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3072.728350] LustreError: 99225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3073.050709] Lustre: server umount lustre-MDT0001 complete [ 3084.460409] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3087.214489] Key type lgssc unregistered [ 3087.530653] LNet: 102781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3087.543155] LNetError: 102781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3087.563416] LNet: Removed LNI 192.168.204.126@tcp [ 3088.487457] Key type .llcrypt unregistered [ 3088.489139] Key type ._llcrypt unregistered [ 3104.409672] Key type ._llcrypt registered [ 3104.412627] Key type .llcrypt registered [ 3104.497715] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3105.229300] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3105.303103] alg: No test for adler32 (adler32-zlib) [ 3106.336334] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 3106.545625] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3108.215595] Key type lgssc registered [ 3109.271910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3114.788733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3114.799630] LustreError: 103864:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3114.821818] LustreError: 103868:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3114.836384] LustreError: 103864:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4974 [ 3114.849486] LustreError: 103864:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3114.865197] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3114.869097] LustreError: 103864:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3116.284913] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.186868] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3123.421963] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3127.912894] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3133.539581] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lsmod [ 3136.402244] LustreError: 105068:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 3136.452411] LustreError: 105075:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 3136.458703] LustreError: 105068:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4954 [ 3136.467206] LustreError: 105068:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3136.475100] LustreError: 105068:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3136.483864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3137.162905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3142.629280] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3142.633248] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3142.644784] Lustre: Skipped 2 previous similar messages [ 3142.658127] Lustre: Skipped 2 previous similar messages [ 3147.745511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3147.749423] Lustre: Skipped 1 previous similar message [ 3148.779663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3148.799729] Lustre: Skipped 2 previous similar messages [ 3153.889625] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3153.903206] Lustre: Skipped 2 previous similar messages [ 3154.399290] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3154.619132] Lustre: server umount lustre-MDT0000 complete [ 3158.021513] LustreError: 103883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787363619 with bad export cookie 2162383967254761767 [ 3158.026704] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3158.036133] LustreError: 103883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3159.019234] LustreError: 103901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3159.019803] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3159.041328] LustreError: 103901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3159.071444] LustreError: 104338:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000b6a479bf x1874186540700672/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3159.079214] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3159.285583] Lustre: server umount lustre-MDT0001 complete [ 3164.769450] Lustre: server umount lustre-OST0000 complete [ 3172.548851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3172.907733] LustreError: 106139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3172.985305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3176.522054] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3178.468985] LustreError: 106140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3182.571745] LustreError: 106139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3184.279513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3184.598879] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3188.878280] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3195.971490] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3201.770500] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3207.695615] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3208.058287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3213.998150] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3217.277827] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3220.934849] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3227.617803] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3227.624831] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3227.640657] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3227.656934] Lustre: Skipped 1 previous similar message [ 3233.333918] Lustre: server umount lustre-OST0000 complete [ 3237.861697] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3237.877832] Lustre: Skipped 2 previous similar messages [ 3243.468694] Lustre: server umount lustre-MDT0000 complete [ 3247.358991] LustreError: 106121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787363709 with bad export cookie 2162383967254762453 [ 3247.364095] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3247.369107] LustreError: 106121:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3247.653186] Lustre: server umount lustre-MDT0001 complete [ 3259.862288] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3262.713692] Key type lgssc unregistered [ 3263.095790] LNet: 109674:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3263.108315] LNetError: 109674:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3263.132670] LNet: Removed LNI 192.168.204.126@tcp [ 3264.211679] Key type .llcrypt unregistered [ 3264.215210] Key type ._llcrypt unregistered [ 3288.058469] Key type ._llcrypt registered [ 3288.060216] Key type .llcrypt registered [ 3288.176743] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3288.834615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3288.853453] alg: No test for adler32 (adler32-zlib) [ 3289.931587] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 3290.145543] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3291.775853] Key type lgssc registered [ 3292.891895] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3300.231368] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3300.269399] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.843120] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3306.205706] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3312.098680] LustreError: 111094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3312.124518] LustreError: 111094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3316.748666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3317.125057] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3321.536924] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3329.185931] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3334.546986] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3341.302574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3341.660834] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3346.820362] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3347.811326] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3355.271881] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3361.087804] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3364.672648] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3370.697214] Lustre: server umount lustre-OST0000 complete [ 3377.635438] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3377.636739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3377.647355] Lustre: Skipped 1 previous similar message [ 3377.661233] Lustre: Skipped 1 previous similar message [ 3382.752465] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.872962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3387.891624] Lustre: Skipped 2 previous similar messages [ 3388.384676] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3388.603074] Lustre: server umount lustre-MDT0000 complete [ 3388.895124] Lustre: 110406:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787363834/real 1787363834] req@000000007dd51e35 x1874186733652352/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787363850 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3388.927186] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3388.952736] LustreError: 111094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3388.966381] LustreError: 111094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3389.797639] Lustre: 110407:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787363835/real 1787363835] req@000000008b3848f7 x1874186733652992/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787363851 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3392.856036] LustreError: 111074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787363854 with bad export cookie 11149460455962944151 [ 3392.862052] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3392.865708] LustreError: 111074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3393.260453] Lustre: server umount lustre-MDT0001 complete [ 3405.905577] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3408.462851] Key type lgssc unregistered [ 3408.744549] LNet: 114734:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3408.759372] LNetError: 114734:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3408.782060] LNet: Removed LNI 192.168.204.126@tcp [ 3409.625223] Key type .llcrypt unregistered [ 3409.627618] Key type ._llcrypt unregistered [ 3425.324952] Key type ._llcrypt registered [ 3425.327225] Key type .llcrypt registered [ 3425.406971] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3426.179855] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3426.196156] alg: No test for adler32 (adler32-zlib) [ 3427.258951] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 3427.497849] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3429.159514] Key type lgssc registered [ 3429.996413] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3438.077473] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3438.124699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3439.586405] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3444.917400] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3449.827394] LustreError: 115957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3449.849103] LustreError: 115957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3454.672232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3454.945610] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 3455.155801] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3460.452353] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3468.721338] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3474.556374] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3481.089970] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3481.408322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.490875] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3488.411888] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3496.081475] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3509.226263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3509.231368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3509.246486] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3510.250280] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3510.250917] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3510.251345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3515.363314] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3515.376320] Lustre: Skipped 2 previous similar messages [ 3520.487220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3520.991154] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3521.159182] Lustre: server umount lustre-OST0000 complete [ 3525.602399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3525.610115] Lustre: Skipped 1 previous similar message [ 3530.367295] Lustre: server umount lustre-MDT0000 complete [ 3530.723795] LustreError: 116949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3530.742461] LustreError: 116949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3533.706489] LustreError: 115940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787363995 with bad export cookie 2064655875211668817 [ 3533.713415] LustreError: 115940:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3533.714447] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3534.088348] Lustre: server umount lustre-MDT0001 complete [ 3544.166856] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3546.864517] Key type lgssc unregistered [ 3547.142586] LNet: 119495:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3547.148359] LNetError: 119495:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3548.199984] LNet: Removed LNI 192.168.204.126@tcp [ 3548.969333] Key type .llcrypt unregistered [ 3548.974109] Key type ._llcrypt unregistered [ 3572.542865] Key type ._llcrypt registered [ 3572.546346] Key type .llcrypt registered [ 3572.650756] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3573.945137] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3574.079375] alg: No test for adler32 (adler32-zlib) [ 3575.139698] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 3575.489227] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3577.199310] Key type lgssc registered [ 3578.522395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3586.097821] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3586.117374] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3587.616284] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3592.291944] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3597.796995] LustreError: 120912:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3597.841118] LustreError: 120912:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3601.266606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3601.603043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3607.041683] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3615.415316] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3620.801466] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3627.456981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3627.826426] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3633.979302] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3634.166975] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3641.033561] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3652.175924] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3656.210194] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3656.220338] Lustre: Skipped 2 previous similar messages [ 3660.926353] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3660.968987] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3661.023331] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3661.033368] Lustre: Skipped 2 previous similar messages [ 3670.119616] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3670.124904] Lustre: Skipped 1 previous similar message [ 3676.499600] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.26@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3676.507492] Lustre: Skipped 2 previous similar messages [ 3678.383288] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.26@tcp 192.168.0.[2,10]@tcp [ 3690.472686] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.26/24@tcp in log lustre-MDT0000 [ 3690.481083] Lustre: Skipped 2 previous similar messages [ 3697.179787] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.26/24@tcp [ 3697.188837] Lustre: Skipped 1 previous similar message [ 3705.824519] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3705.836423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3705.852703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3707.876148] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3707.886304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3707.932226] LustreError: 122749:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3707.946136] LustreError: 122749:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3708.055996] Lustre: server umount lustre-OST0000 complete [ 3712.993974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3713.000928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3713.006037] Lustre: Skipped 1 previous similar message [ 3713.013530] Lustre: Skipped 1 previous similar message [ 3718.112904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3718.125599] Lustre: Skipped 1 previous similar message [ 3723.233218] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3723.247117] Lustre: Skipped 1 previous similar message [ 3726.303172] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3726.485313] Lustre: server umount lustre-MDT0000 complete [ 3727.329602] LustreError: 120906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3727.357551] LustreError: 120906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3730.745528] LustreError: 123559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787364192 with bad export cookie 8184205557562517211 [ 3730.752769] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3730.758037] LustreError: 123559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3731.340697] Lustre: server umount lustre-MDT0001 complete [ 3742.179858] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3744.823828] Key type lgssc unregistered [ 3745.121235] LNet: 125976:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3745.131994] LNetError: 125976:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3745.147176] LNet: Removed LNI 192.168.204.126@tcp [ 3745.950698] Key type .llcrypt unregistered [ 3745.952449] Key type ._llcrypt unregistered [ 3776.181400] Key type ._llcrypt registered [ 3776.185217] Key type .llcrypt registered [ 3776.342850] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3777.802948] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3777.846804] alg: No test for adler32 (adler32-zlib) [ 3779.021368] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 3779.338933] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3781.019222] Key type lgssc registered [ 3782.332537] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3790.250116] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3790.281633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3791.730872] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3791.761485] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.26/24@tcp [ 3791.773117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3796.054587] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3802.084934] LustreError: 127537:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3802.097745] LustreError: 127537:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3805.361248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3805.693831] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3805.733663] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.26/24@tcp [ 3805.790022] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3810.489542] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3817.937862] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3824.013431] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3830.637547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3830.947856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3836.871473] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 3837.100543] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3844.730111] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3860.968805] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3860.975706] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3860.985115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3860.996352] Lustre: Skipped 1 previous similar message [ 3861.021187] Lustre: Skipped 1 previous similar message [ 3865.088423] Lustre: server umount lustre-OST0000 complete [ 3871.203103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3871.207859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3871.227635] Lustre: Skipped 1 previous similar message [ 3875.773559] Lustre: server umount lustre-MDT0000 complete [ 3876.321145] LustreError: 127536:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3876.343248] LustreError: 127536:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3879.613620] LustreError: 130373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787364341 with bad export cookie 17522889499968948561 [ 3879.614738] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3879.627800] LustreError: 130373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3880.012529] Lustre: server umount lustre-MDT0001 complete [ 3891.087277] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 3894.128525] Key type lgssc unregistered [ 3894.533976] LNet: 131128:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3894.546436] LNetError: 131128:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3895.596869] LNet: Removed LNI 192.168.204.126@tcp [ 3896.623633] Key type .llcrypt unregistered [ 3896.625207] Key type ._llcrypt unregistered [ 3921.504692] Key type ._llcrypt registered [ 3921.507078] Key type .llcrypt registered [ 3921.590353] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 3936.536144] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 3937.700685] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3937.771948] alg: No test for adler32 (adler32-zlib) [ 3938.878537] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 3939.141100] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 3940.823252] Key type lgssc registered [ 3942.044118] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3980.330611] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3980.377704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3981.739508] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3981.790295] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3981.922465] Lustre: lustre-MDT0000: new disk, initializing [ 3982.034570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3982.066698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3987.465126] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4001.134614] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4001.278655] Lustre: 134533:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4001.411087] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4001.417705] Lustre: Skipped 1 previous similar message [ 4001.510886] Lustre: lustre-MDT0001: new disk, initializing [ 4001.676415] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4001.728318] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4001.737960] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4007.311607] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4017.116640] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4023.664604] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4030.888338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4031.209655] Lustre: lustre-OST0000: new disk, initializing [ 4031.214525] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4031.224808] Lustre: 136501:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4031.313908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4037.745429] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4038.223695] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4038.231826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4038.368240] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4047.003465] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4051.701555] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4051.937931] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4056.872847] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4057.060699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4059.111196] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4059.122521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4059.123438] Lustre: Skipped 1 previous similar message [ 4059.143769] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4064.234838] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4064.244429] Lustre: Skipped 2 previous similar messages [ 4065.249294] Lustre: server umount lustre-OST0000 complete [ 4069.348064] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4069.351907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4069.356527] Lustre: Skipped 1 previous similar message [ 4074.466181] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4074.477519] Lustre: Skipped 2 previous similar messages [ 4079.587167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4079.602455] Lustre: Skipped 1 previous similar message [ 4083.167198] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4083.513532] Lustre: server umount lustre-MDT0000 complete [ 4087.624310] LustreError: 134523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787364549 with bad export cookie 7776493957932347896 [ 4087.627963] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4087.634364] LustreError: 134523:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4087.932166] Lustre: server umount lustre-MDT0001 complete [ 4097.769953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4098.299204] LustreError: 138723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4098.313995] LustreError: 138723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4098.365900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4103.650988] LustreError: 138724:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4103.720295] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4108.773433] LustreError: 138723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4112.697734] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4113.147907] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4118.178213] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4126.357988] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4132.750199] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4140.356425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4140.875286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4149.373545] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4159.170543] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4164.896454] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4165.063862] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4170.407502] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4170.706453] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4182.155774] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4182.418464] Lustre: lustre-OST0001: new disk, initializing [ 4182.425167] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4182.435237] Lustre: 142095:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4182.584126] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4190.543114] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4191.807512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4191.823366] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4191.932971] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4200.709232] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4201.103541] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4207.700280] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4208.079850] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4215.039382] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b3a44402800.ost_server_uuid 50 [ 4217.014219] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b3a44402800.ost_server_uuid in IDLE state after 0 sec [ 4228.064596] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4228.080766] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4228.083275] LustreError: Skipped 1 previous similar message [ 4228.088348] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4228.104456] Lustre: Skipped 1 previous similar message [ 4228.945604] Lustre: server umount lustre-OST0001 complete [ 4233.212266] LustreError: 140545:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4233.246470] LustreError: 140545:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4234.208749] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4234.230285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4236.271878] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4238.310458] LustreError: 142308:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4238.330530] LustreError: 142308:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4239.829419] Lustre: server umount lustre-OST0000 complete [ 4246.496623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4246.504775] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4246.523860] Lustre: Skipped 1 previous similar message [ 4246.529131] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4246.534313] Lustre: Skipped 5 previous similar messages [ 4250.317168] Lustre: server umount lustre-MDT0000 complete [ 4253.676911] LustreError: 139451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4253.711045] LustreError: 139451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4254.484427] LustreError: 142094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787364716 with bad export cookie 7776493957932348785 [ 4254.503553] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4254.511000] LustreError: 142094:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4255.020787] Lustre: server umount lustre-MDT0001 complete [ 4279.507392] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4282.745349] Key type lgssc unregistered [ 4283.123627] LNet: 145106:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4283.139099] LNetError: 145106:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4283.159197] LNet: Removed LNI 192.168.204.126@tcp [ 4284.389209] Key type .llcrypt unregistered [ 4284.391215] Key type ._llcrypt unregistered [ 4330.487610] Key type ._llcrypt registered [ 4330.490189] Key type .llcrypt registered [ 4330.593678] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4347.676942] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4348.917978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4349.021243] alg: No test for adler32 (adler32-zlib) [ 4350.110385] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 4350.447389] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4352.263162] Key type lgssc registered [ 4353.591925] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4399.248849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4399.288672] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4400.613910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4400.672623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4400.759883] Lustre: lustre-MDT0000: new disk, initializing [ 4400.874316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4400.902891] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4405.954687] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4417.828708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4418.025839] Lustre: 149264:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4418.100935] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4418.105471] Lustre: Skipped 1 previous similar message [ 4418.227060] Lustre: lustre-MDT0001: new disk, initializing [ 4418.419922] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4418.469376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4418.485164] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4423.290525] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4434.997947] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4443.374623] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4451.884767] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4452.196366] Lustre: lustre-OST0000: new disk, initializing [ 4452.201198] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4452.209555] Lustre: 151231:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4452.384065] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4461.417808] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4461.652091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4461.667653] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4461.842472] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4472.025235] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4477.663544] Lustre: Failing over lustre-OST0000 [ 4478.042285] Lustre: server umount lustre-OST0000 complete [ 4495.823295] Lustre: 147486:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787364941/real 1787364941] req@000000004302fb58 x1874187845143808/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787364957 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4495.867233] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.332643] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4498.910860] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4498.947406] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4498.988975] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4499.935134] Lustre: 147487:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787364944/real 1787364944] req@000000008dd9df74 x1874187845144832/t0(0) o13->lustre-OST0000-osc-MDT0000@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787364960 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4499.977237] Lustre: 147487:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4500.976299] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4504.095503] Lustre: 147485:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787364949/real 1787364949] req@0000000089bb9a5a x1874187845145984/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787364965 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4506.717449] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4506.732109] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 4506.733227] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4508.831274] Lustre: 147485:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787364954/real 1787364954] req@000000006d08b5ae x1874187845147136/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787364970 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4508.860686] Lustre: 147485:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4511.633207] Lustre: Failing over lustre-MDT0000 [ 4511.804344] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 4512.054615] Lustre: server umount lustre-MDT0000 complete [ 4515.809099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4515.818103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4521.455366] LustreError: 151405:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4521.504398] LustreError: 151405:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4522.012049] LustreError: 153009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4526.577957] LustreError: 149274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4526.592549] LustreError: 149274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4531.691875] LustreError: 149271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4531.705650] LustreError: 149271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4532.181817] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4532.409655] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4532.760087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4532.826209] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4533.356981] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4537.834314] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4537.849085] Lustre: Skipped 1 previous similar message [ 4537.866781] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4537.916309] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4538.130410] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4548.077348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4548.078525] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4548.114769] Lustre: Skipped 2 previous similar messages [ 4548.120805] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4548.134360] Lustre: Skipped 1 previous similar message [ 4553.191670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4553.194957] Lustre: Skipped 1 previous similar message [ 4554.164906] Lustre: server umount lustre-OST0000 complete [ 4563.425231] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4563.433848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4563.450705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4563.467684] Lustre: Skipped 2 previous similar messages [ 4563.490718] Lustre: Skipped 1 previous similar message [ 4565.763792] Lustre: server umount lustre-MDT0000 complete [ 4568.545508] LustreError: 149274:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4568.555270] LustreError: 149274:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4570.558133] LustreError: 149254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787365032 with bad export cookie 6182069617936233720 [ 4570.559457] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4570.567047] LustreError: 149254:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4570.922767] Lustre: server umount lustre-MDT0001 complete [ 4584.509147] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 4587.534232] Key type lgssc unregistered [ 4587.885755] LNet: 154892:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4587.899121] LNetError: 154892:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4587.911956] LNet: Removed LNI 192.168.204.126@tcp [ 4589.090574] Key type .llcrypt unregistered [ 4589.096983] Key type ._llcrypt unregistered [ 4630.623644] Key type ._llcrypt registered [ 4630.632200] Key type .llcrypt registered [ 4630.752737] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4632.555036] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4632.601983] alg: No test for adler32 (adler32-zlib) [ 4633.668960] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 4633.970813] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 4635.711269] Key type lgssc registered [ 4637.065872] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4646.465459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4646.482030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.871761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4653.576666] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4658.147072] LustreError: 156416:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4658.164842] LustreError: 156416:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4663.272449] LustreError: 156417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4663.796059] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4667.622325] LustreError: 156445:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4667.635707] LustreError: 156445:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4667.646635] LustreError: 156445:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4668.107868] Lustre: server umount lustre-MDT0000 complete [ 4685.224846] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 4696.291811] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 4740.420164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4740.923075] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4741.004908] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4741.084239] Lustre: lustre-MDT0000: new disk, initializing [ 4741.239969] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4741.276793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4746.748897] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4760.612509] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4760.782453] Lustre: 159751:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4760.841257] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4760.843933] Lustre: Skipped 1 previous similar message [ 4760.900304] Lustre: lustre-MDT0001: new disk, initializing [ 4760.982227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4761.034209] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4761.050333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4766.813868] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4778.480791] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4785.279777] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4793.321690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4793.579648] Lustre: lustre-OST0000: new disk, initializing [ 4793.582458] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4793.587549] Lustre: 161718:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4793.696464] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4799.034174] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4799.039837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4799.159097] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4800.542397] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 4811.924604] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4816.855502] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4818.943949] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 22:21:17 (1787365277) [ 5104.097374] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5104.108271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5104.134808] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5106.658026] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5106.658348] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5106.663209] Lustre: Skipped 1 previous similar message [ 5106.693765] Lustre: Skipped 1 previous similar message [ 5111.776386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5111.792822] Lustre: Skipped 3 previous similar messages [ 5116.383222] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5116.896479] LustreError: 163500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5116.940593] LustreError: 163500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5118.622427] Lustre: server umount lustre-MDT0000 complete [ 5122.019470] LustreError: 159755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5122.035148] LustreError: 159755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5122.440503] LustreError: 159741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787365584 with bad export cookie 3902083370781251458 [ 5122.445441] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5122.456098] LustreError: 159741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5127.135647] LustreError: 159756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5127.139245] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5127.189156] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5129.027169] Lustre: server umount lustre-MDT0001 complete [ 5139.634280] Lustre: server umount lustre-OST0000 complete [ 5150.532306] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5161.107437] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5202.136733] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5202.420595] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5202.450668] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5202.573989] Lustre: lustre-MDT0000: new disk, initializing [ 5202.646412] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5202.658715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5207.549308] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5221.661234] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5221.895431] Lustre: 166391:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5221.941558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5221.950402] Lustre: Skipped 1 previous similar message [ 5222.145272] Lustre: lustre-MDT0001: new disk, initializing [ 5222.379087] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5222.462185] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5222.482044] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5228.310258] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5238.509824] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5244.673610] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5252.662787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5252.976446] Lustre: lustre-OST0000: new disk, initializing [ 5252.983187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5252.990104] Lustre: 168358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5253.076526] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5259.955742] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5262.964458] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5262.989391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5263.115437] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5269.907910] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5275.283241] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5275.525392] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5280.995939] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5281.360716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5283.821192] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5283.823928] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5283.839520] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5283.870417] Lustre: Skipped 1 previous similar message [ 5289.672076] Lustre: server umount lustre-OST0000 complete [ 5294.048629] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5294.051367] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5294.063707] LustreError: Skipped 1 previous similar message [ 5294.076810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5294.083971] Lustre: Skipped 4 previous similar messages [ 5299.872484] Lustre: server umount lustre-MDT0000 complete [ 5303.721067] LustreError: 166384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787365765 with bad export cookie 3902083370781410281 [ 5303.724945] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5303.738176] LustreError: 166384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5304.052657] Lustre: server umount lustre-MDT0001 complete [ 5312.666331] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:29:31 (1787365771) [ 5330.587809] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5341.213852] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5385.338704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5385.766468] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5385.824257] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5385.945073] Lustre: lustre-MDT0000: new disk, initializing [ 5386.087292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5386.113391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5390.942689] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5402.502221] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5402.650674] Lustre: 173119:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5403.004092] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5407.733889] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5417.733331] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5425.031352] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5433.313102] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5433.889094] Lustre: lustre-OST0000: new disk, initializing [ 5433.894321] Lustre: Skipped 1 previous similar message [ 5433.901460] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5433.917469] Lustre: Skipped 2 previous similar messages [ 5433.928113] Lustre: 175095:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5439.668484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5439.678053] Lustre: Skipped 1 previous similar message [ 5439.695966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5439.815465] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5442.046427] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5454.685667] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5465.579670] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5465.580909] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5465.589670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5465.589679] Lustre: Skipped 2 previous similar messages [ 5465.598711] LustreError: Skipped 1 previous similar message [ 5465.648794] Lustre: Skipped 2 previous similar messages [ 5467.303695] Lustre: server umount lustre-OST0000 complete [ 5474.788218] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5477.959650] Lustre: server umount lustre-MDT0000 complete [ 5480.932722] LustreError: 173125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5480.945843] LustreError: 173125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5482.046235] LustreError: 173918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787365943 with bad export cookie 3902083370781411170 [ 5482.049595] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5482.059246] LustreError: 173918:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5482.388056] Lustre: server umount lustre-MDT0001 complete [ 5491.789537] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 22:32:30 (1787365950) [ 5510.810699] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 5522.928726] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5561.656641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5562.013310] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5562.042848] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5562.226504] Lustre: lustre-MDT0000: new disk, initializing [ 5562.382265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5562.401542] Lustre: Skipped 2 previous similar messages [ 5562.445986] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5567.996971] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5580.784488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5581.044584] Lustre: 179422:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5581.629751] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5587.234181] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5598.459327] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5604.914455] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5612.742384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5613.178514] Lustre: 181398:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5619.834826] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5621.325639] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5621.479719] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5630.367636] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5642.210916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5642.225683] Lustre: Skipped 2 previous similar messages [ 5642.226063] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5642.239083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5642.252303] Lustre: Skipped 5 previous similar messages [ 5643.615634] Lustre: server umount lustre-OST0000 complete [ 5654.174116] Lustre: server umount lustre-MDT0000 complete [ 5657.578231] LustreError: 180596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5657.606635] LustreError: 180596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5657.925111] LustreError: 179412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787366119 with bad export cookie 3902083370781412255 [ 5657.932775] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5657.935054] LustreError: 179412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5658.415972] Lustre: server umount lustre-MDT0001 complete [ 5671.085353] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5673.916653] Key type lgssc unregistered [ 5674.314905] LNet: 183260:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5674.331184] LNetError: 183260:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5675.365726] LNet: Removed LNI 192.168.204.126@tcp [ 5676.180586] Key type .llcrypt unregistered [ 5676.184422] Key type ._llcrypt unregistered [ 5704.810426] Key type ._llcrypt registered [ 5704.815374] Key type .llcrypt registered [ 5705.008748] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5705.996491] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5706.160180] alg: No test for adler32 (adler32-zlib) [ 5707.239548] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 5707.479072] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5709.199191] Key type lgssc registered [ 5710.354793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5718.352141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5718.370117] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5719.882581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5724.813485] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5729.257571] LustreError: 184676:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5729.269718] LustreError: 184676:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5734.374282] LustreError: 184675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5734.394292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5734.723791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5738.968617] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5747.575579] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5754.422118] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5761.394834] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5761.698947] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5765.820516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5767.845643] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5774.479127] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5801.952185] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5801.953333] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5801.958966] LustreError: Skipped 1 previous similar message [ 5801.960405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5801.977106] Lustre: Skipped 1 previous similar message [ 5801.994480] Lustre: Skipped 1 previous similar message [ 5803.108595] Lustre: server umount lustre-OST0000 complete [ 5811.680302] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5811.687855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5811.701725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5813.591840] Lustre: server umount lustre-MDT0000 complete [ 5817.313576] LustreError: 187028:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5817.330127] LustreError: 187028:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5818.127453] LustreError: 187180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787366279 with bad export cookie 16859525606266423934 [ 5818.139839] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5818.142841] LustreError: 187180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5818.603985] Lustre: server umount lustre-MDT0001 complete [ 5830.534393] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 5833.604081] Key type lgssc unregistered [ 5833.918603] LNet: 188239:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5833.924434] LNetError: 188239:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5833.937130] LNet: Removed LNI 192.168.204.126@tcp [ 5834.855282] Key type .llcrypt unregistered [ 5834.857280] Key type ._llcrypt unregistered [ 5863.817027] Key type ._llcrypt registered [ 5863.819366] Key type .llcrypt registered [ 5863.934334] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 5865.539616] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5865.592957] alg: No test for adler32 (adler32-zlib) [ 5866.803988] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 5867.128557] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 5868.815174] Key type lgssc registered [ 5870.055926] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5878.338360] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5878.362620] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5879.850163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5884.388145] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5890.019739] LustreError: 189659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5890.055122] LustreError: 189659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5893.192342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5893.673677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5898.934083] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5907.367699] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5913.992431] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5922.107575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5922.468793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5924.530321] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5929.329678] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 5938.141599] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5946.470692] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 5950.435899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5950.442068] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5950.465423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5951.532558] Lustre: lustre-OST0000: Not available for connect from 192.168.204.26@tcp (stopping) [ 5951.542102] Lustre: Skipped 1 previous similar message [ 5952.355625] Lustre: server umount lustre-OST0000 complete [ 5960.818393] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8b3a49559000.ost_server_uuid 50 [ 5962.571304] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5977.341500] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.350477] Lustre: Skipped 1 previous similar message [ 5977.357982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5977.365801] Lustre: Skipped 1 previous similar message [ 5982.692513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5982.698341] Lustre: Skipped 1 previous similar message [ 5983.671360] Lustre: server umount lustre-MDT0000 complete [ 5987.190448] LustreError: 189638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787366449 with bad export cookie 16330796847110648478 [ 5987.197687] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5987.519651] Lustre: server umount lustre-MDT0001 complete [ 5995.260305] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 22:40:54 (1787366454) [ 6000.900178] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6001.271729] LustreError: 193571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6001.330841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6005.553853] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6006.757221] LustreError: 193572:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6011.880504] LustreError: 193571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6013.418370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6013.837384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6018.167848] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6026.070716] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6031.608356] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6038.116335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6038.495977] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6044.696530] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6046.733884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6052.964991] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6060.427941] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6060.815140] Lustre: lustre-OST0001: new disk, initializing [ 6060.827423] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6060.840579] Lustre: 196412:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6060.985120] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6068.382878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6068.395897] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6068.554369] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6068.962953] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6080.277635] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6084.085789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6084.101687] Lustre: Skipped 1 previous similar message [ 6084.106917] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6084.119035] Lustre: Skipped 1 previous similar message [ 6097.375277] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6097.481917] Lustre: server umount lustre-OST0000 complete [ 6099.437045] LustreError: 195394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6099.455062] LustreError: 195394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6103.729639] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6103.931920] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6108.877795] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6109.161906] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6109.675149] LustreError: 195394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6109.708143] LustreError: 195394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 6125.034669] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6125.036606] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6125.046640] LustreError: Skipped 1 previous similar message [ 6125.051976] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6125.056412] Lustre: Skipped 1 previous similar message [ 6125.068769] Lustre: Skipped 8 previous similar messages [ 6129.272934] Lustre: server umount lustre-OST0001 complete [ 6135.264704] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6135.284492] Lustre: Skipped 1 previous similar message [ 6139.883756] Lustre: server umount lustre-MDT0000 complete [ 6140.385647] LustreError: 193571:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6140.405359] LustreError: 193571:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6143.348472] LustreError: 194305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787366605 with bad export cookie 16330796847110649542 [ 6143.353098] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6143.366227] LustreError: 194305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6143.762541] Lustre: server umount lustre-MDT0001 complete [ 6169.810369] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 22:43:48 (1787366628) [ 6177.480090] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6177.867719] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6177.912464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6178.183119] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6183.096320] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6191.872625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6192.005849] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6192.074281] Lustre: 199858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6197.054656] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6204.771670] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6210.721392] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6218.930858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6219.307354] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6225.562251] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6227.074817] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6236.668491] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6244.034142] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6244.122933] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6244.387438] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6244.390357] Lustre: Skipped 2 previous similar messages [ 6251.631970] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6259.670974] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6263.277661] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6263.295716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6263.305634] Lustre: Skipped 3 previous similar messages [ 6269.044577] Lustre: server umount lustre-OST0000 complete [ 6270.495206] LustreError: 202753:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6288.867641] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6288.868421] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6288.885158] Lustre: Skipped 1 previous similar message [ 6293.775542] Lustre: server umount lustre-OST0001 complete [ 6304.281204] Lustre: server umount lustre-MDT0000 complete [ 6308.088959] LustreError: 201729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787366769 with bad export cookie 16330796847110650956 [ 6308.093657] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6308.101728] LustreError: 201729:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6308.383895] Lustre: server umount lustre-MDT0001 complete [ 6331.771896] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 22:46:30 (1787366790) [ 6347.613521] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 6357.178832] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 6395.482300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6395.786325] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6395.853314] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6395.946727] Lustre: lustre-MDT0000: new disk, initializing [ 6396.007447] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6396.042649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6400.701979] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6412.011092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6412.131433] Lustre: 207954:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6412.218926] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6412.223736] Lustre: Skipped 1 previous similar message [ 6412.336081] Lustre: lustre-MDT0001: new disk, initializing [ 6412.457398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6412.469866] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6417.010293] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6427.759748] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6434.958906] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6442.440280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6442.841512] Lustre: lustre-OST0000: new disk, initializing [ 6442.844519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6442.851356] Lustre: 209934:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6449.751132] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6449.763489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6449.908988] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6450.114647] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6460.265637] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6465.592632] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6465.868254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6471.237631] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6471.490421] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6475.744468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6475.753270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6475.765110] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6475.773615] Lustre: Skipped 4 previous similar messages [ 6475.784810] Lustre: Skipped 8 previous similar messages [ 6479.858303] Lustre: server umount lustre-OST0000 complete [ 6484.449117] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6489.958747] Lustre: server umount lustre-MDT0000 complete [ 6491.110090] LustreError: 207963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6491.129130] LustreError: 207963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6494.244624] LustreError: 207947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787366956 with bad export cookie 16330796847110652517 [ 6494.251267] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6494.258513] LustreError: 207947:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6494.541630] Lustre: server umount lustre-MDT0001 complete [ 6505.556135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6511.156591] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6520.395493] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6524.665277] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6533.130225] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6541.116387] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6549.231556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6555.981125] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6563.799843] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6569.282808] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6569.592699] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6574.155664] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6574.431572] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6581.229781] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6581.231123] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6581.241505] LustreError: Skipped 1 previous similar message [ 6581.268072] Lustre: Skipped 3 previous similar messages [ 6582.965131] Lustre: server umount lustre-OST0000 complete [ 6589.791549] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6595.152587] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6624.165282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6633.138515] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6642.067304] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6644.360395] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6646.908849] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6646.929737] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6646.943170] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6652.385883] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6652.404113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6652.407920] Lustre: Skipped 5 previous similar messages [ 6656.272510] Lustre: server umount lustre-OST0000 complete [ 6666.971870] Lustre: server umount lustre-MDT0000 complete [ 6667.745351] LustreError: 212153:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6667.774881] LustreError: 212153:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6671.532866] LustreError: 212134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367133 with bad export cookie 16330796847110653399 [ 6671.542931] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6671.544070] LustreError: 212134:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6671.968131] Lustre: server umount lustre-MDT0001 complete [ 6682.060916] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 22:52:21 (1787367141) [ 6689.036708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6689.454485] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6689.459829] Lustre: Skipped 6 previous similar messages [ 6694.640404] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6703.954500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6709.420338] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6716.681732] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6723.305911] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6730.429461] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6737.541561] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6748.623494] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6755.441376] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6755.718297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6761.479646] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6761.719711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6769.928374] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6770.095204] Lustre: lustre-OST0001: new disk, initializing [ 6770.098155] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6770.102630] Lustre: 221493:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6776.418670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6776.429457] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6776.575157] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6776.728431] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6786.246156] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6791.498752] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6791.714352] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6797.490461] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6797.701878] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6802.414919] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6802.430261] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6802.442848] Lustre: Skipped 5 previous similar messages [ 6802.473123] LustreError: Skipped 1 previous similar message [ 6805.987121] Lustre: server umount lustre-OST0001 complete [ 6812.029682] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6812.237975] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6816.470293] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6816.755488] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6845.722061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6851.072494] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6851.087873] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6851.088425] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6851.123693] Lustre: Skipped 2 previous similar messages [ 6852.112401] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6859.787738] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6861.442531] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6869.462522] Lustre: server umount lustre-OST0001 complete [ 6892.494663] Lustre: server umount lustre-OST0000 complete [ 6898.656636] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6898.662044] LustreError: Skipped 2 previous similar messages [ 6903.283719] Lustre: server umount lustre-MDT0000 complete [ 6907.789808] LustreError: 218101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367369 with bad export cookie 16330796847110654358 [ 6907.796122] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6907.810931] LustreError: 218101:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6922.208602] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6922.510880] Lustre: server umount lustre-MDT0001 complete [ 6947.442925] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 22:56:46 (1787367406) [ 6954.851310] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6955.202337] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6955.226741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6960.302386] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6970.635709] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.803641] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6970.839882] Lustre: 226735:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6975.625577] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 6983.533276] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6989.337844] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6996.135959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6996.334196] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7002.522540] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7010.374735] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7021.050219] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7021.201152] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7028.050899] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7036.923657] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7038.901248] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7043.750408] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7044.014169] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7049.132828] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7049.525475] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7056.307489] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8b3a4955a000.ost_server_uuid 50 [ 7058.394284] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8b3a4955a000.ost_server_uuid in IDLE state after 0 sec [ 7059.884501] Lustre: Permanently deactivating lustre-OST0001 [ 7059.900224] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7064.011384] Lustre: Permanently reactivating lustre-OST0001 [ 7064.043258] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7064.048889] Lustre: Skipped 2 previous similar messages [ 7070.189375] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7070.198716] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7070.214224] Lustre: Skipped 6 previous similar messages [ 7070.225257] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7070.230818] Lustre: Skipped 14 previous similar messages [ 7074.515976] Lustre: server umount lustre-OST0001 complete [ 7075.827506] LustreError: 229126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7075.858742] LustreError: 229126:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 38 previous similar messages [ 7085.387925] Lustre: server umount lustre-OST0000 complete [ 7095.515629] Lustre: server umount lustre-MDT0000 complete [ 7099.705810] LustreError: 226726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367561 with bad export cookie 16330796847110655653 [ 7099.724627] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7099.725895] LustreError: 226726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7100.311876] Lustre: server umount lustre-MDT0001 complete [ 7128.576658] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 22:59:47 (1787367587) [ 7136.827341] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7137.055599] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7137.133944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7137.141520] Lustre: Skipped 2 previous similar messages [ 7142.330113] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7151.283799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7151.418853] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7151.486861] Lustre: 233435:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7156.641662] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7165.815504] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7172.355888] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7179.977083] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7180.275158] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7180.306894] Lustre: Permanently deactivating lustre-OST0000 [ 7180.323891] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7187.825702] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7188.051720] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7188.177502] LustreError: 235683:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -108 [ 7196.551831] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7204.777859] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7204.848642] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7205.129921] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7205.139408] Lustre: Skipped 11 previous similar messages [ 7212.202592] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7214.810880] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7220.421284] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7223.020501] Lustre: Permanently reactivating lustre-OST0000 [ 7223.048982] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7223.053343] Lustre: Skipped 2 previous similar messages [ 7228.563090] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7228.566637] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 40 seconds. I think it's dead, and I am evicting it. exp 0000000017593b51, cur 1787367690 deadline 1787367680 last 1787367650 [ 7228.576048] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7228.576786] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7228.675332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7243.950837] Lustre: server umount lustre-OST0001 complete [ 7263.201022] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7263.365801] Lustre: server umount lustre-OST0000 complete [ 7274.646199] Lustre: server umount lustre-MDT0000 complete [ 7278.479490] LustreError: 233426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367740 with bad export cookie 16330796847110657326 [ 7278.485821] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7278.489772] LustreError: 233426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7278.861551] Lustre: server umount lustre-MDT0001 complete [ 7289.396607] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7292.643177] Key type lgssc unregistered [ 7293.106740] LNet: 238331:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7293.118640] LNetError: 238331:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7293.136467] LNet: Removed LNI 192.168.204.126@tcp [ 7294.190261] Key type .llcrypt unregistered [ 7294.191802] Key type ._llcrypt unregistered [ 7315.984326] Key type ._llcrypt registered [ 7315.986939] Key type .llcrypt registered [ 7316.101493] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7317.227151] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7317.279154] alg: No test for adler32 (adler32-zlib) [ 7318.361746] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 7318.594703] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7320.239804] Key type lgssc registered [ 7321.516110] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7331.840161] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7331.864269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7333.302580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7337.338330] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7343.589836] LustreError: 239800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7343.613880] LustreError: 239800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7345.430415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7345.631201] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7345.692756] Lustre: 239788:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7345.733354] Lustre: Permanently deactivating lustre-MDT0001 [ 7345.766272] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7346.132549] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7348.574506] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7348.585493] LustreError: 239830:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7348.603743] LustreError: 239830:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7348.617843] LustreError: 239830:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 1, failed: rc = -108 [ 7350.930497] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7359.422641] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7365.650817] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7372.958195] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7373.370907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7378.472470] LustreError: 241644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7378.481942] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7381.145505] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7383.528436] LustreError: 241644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7388.645600] LustreError: 241644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7390.175668] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7395.817932] LustreError: 241645:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7395.835541] LustreError: 241645:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 7398.342137] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7398.629718] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7404.033990] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7404.049375] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7405.647938] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7415.052643] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7419.004500] Lustre: Permanently reactivating lustre-MDT0001 [ 7419.016176] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7419.020624] Lustre: Skipped 2 previous similar messages [ 7421.396713] Lustre: lustre-MDT0001: Client 64353e68-9ec5-439e-87fc-b95c2e7a8880 (at 192.168.204.26@tcp) reconnecting [ 7427.998870] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7429.768162] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7436.931625] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7437.154526] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7442.194436] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7442.428468] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7445.736816] Lustre: Permanently deactivating lustre-MDT0001 [ 7445.756641] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7445.765659] Lustre: Skipped 2 previous similar messages [ 7454.917395] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7462.524769] Lustre: server umount lustre-MDT0000 complete [ 7463.908355] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7463.923518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7463.955435] LustreError: 239800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7464.428550] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7464.444642] Lustre: Skipped 1 previous similar message [ 7466.413904] LustreError: 243350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787367928 with bad export cookie 8713534942573442031 [ 7466.420652] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7466.430721] LustreError: 243350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7466.712125] Lustre: server umount lustre-MDT0001 complete [ 7472.737624] Lustre: server umount lustre-OST0000 complete [ 7485.100324] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 23:05:44 (1787367944) [ 7494.367431] Lustre: server umount lustre-OST0001 complete [ 7501.224172] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 7512.805605] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7556.491641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7556.825571] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7556.832761] Lustre: Skipped 2 previous similar messages [ 7556.867774] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7556.951731] Lustre: lustre-MDT0000: new disk, initializing [ 7557.073575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7557.097767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7561.952337] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7573.972713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7574.112985] Lustre: 248061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7574.172370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7574.178236] Lustre: Skipped 1 previous similar message [ 7574.256446] Lustre: lustre-MDT0001: new disk, initializing [ 7574.354454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7574.444865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7574.462707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7579.587140] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7591.029385] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7598.557492] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7605.882807] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7606.280252] Lustre: lustre-OST0000: new disk, initializing [ 7606.291837] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7606.307212] Lustre: 250027:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7606.398980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7613.003406] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7616.634603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7616.657536] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7616.804488] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7623.133566] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7625.245282] LustreError: 248067:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7633.585263] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7633.717641] Lustre: lustre-OST0001: new disk, initializing [ 7633.721383] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7633.729163] Lustre: 251147:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7635.315733] LustreError: 248067:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7635.337729] LustreError: 251052:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7640.369790] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7642.280676] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7642.299091] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7642.432374] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7645.367181] LustreError: 251052:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7651.338474] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7652.992285] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7657.556931] Lustre: lustre-OST0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 7657.968369] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7657.971619] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7657.982918] LustreError: Skipped 1 previous similar message [ 7658.028512] Lustre: Skipped 2 previous similar messages [ 7661.267509] Lustre: server umount lustre-OST0001 complete [ 7662.649535] LustreError: 250013:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7662.668301] LustreError: 250013:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7683.552515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7683.553546] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7683.563653] LustreError: Skipped 1 previous similar message [ 7683.573202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7683.573211] Lustre: Skipped 2 previous similar messages [ 7684.796698] Lustre: server umount lustre-OST0000 complete [ 7692.263227] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7692.274549] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7692.292275] Lustre: Skipped 1 previous similar message [ 7692.304866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7692.310917] Lustre: Skipped 1 previous similar message [ 7695.270102] Lustre: server umount lustre-MDT0000 complete [ 7698.914283] LustreError: 249263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7698.943089] LustreError: 249263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 11 previous similar messages [ 7699.412064] LustreError: 248051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368161 with bad export cookie 8713534942573444621 [ 7699.421787] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7706.046762] Lustre: server umount lustre-MDT0001 complete [ 7717.658881] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 7720.554647] Key type lgssc unregistered [ 7720.837774] LNet: 253186:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7720.847202] LNetError: 253186:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7720.873670] LNet: Removed LNI 192.168.204.126@tcp [ 7721.670874] Key type .llcrypt unregistered [ 7721.676082] Key type ._llcrypt unregistered [ 7761.425684] Key type ._llcrypt registered [ 7761.429388] Key type .llcrypt registered [ 7761.591692] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 7762.945894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7762.968501] alg: No test for adler32 (adler32-zlib) [ 7763.973287] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 7764.178658] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 7765.831272] Key type lgssc registered [ 7767.524523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7777.247237] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7777.277963] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7778.547479] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7778.586662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7778.739382] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7785.074086] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7794.540250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7794.683774] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7794.714496] Lustre: 255392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7794.948924] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7799.262249] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7807.975176] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7814.081969] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7820.563661] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7820.698637] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7820.818728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7827.047610] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7829.709301] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7835.236316] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7886.306400] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7886.316397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7886.318253] Lustre: Skipped 1 previous similar message [ 7886.321469] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7886.336903] Lustre: Skipped 1 previous similar message [ 7891.427371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7891.431825] Lustre: Skipped 1 previous similar message [ 7892.544388] Lustre: server umount lustre-OST0000 complete [ 7896.643845] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7907.296695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7907.681400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7914.203878] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 7916.907330] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7916.940018] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7916.940396] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7916.972490] Lustre: Skipped 1 previous similar message [ 7921.399339] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7963.134703] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7963.138643] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7963.143627] LustreError: Skipped 1 previous similar message [ 7963.145197] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7963.189809] Lustre: Skipped 1 previous similar message [ 7967.488075] Lustre: server umount lustre-OST0000 complete [ 7973.346454] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7973.351400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7973.368532] Lustre: Skipped 1 previous similar message [ 7973.382921] Lustre: Skipped 2 previous similar messages [ 7978.464967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7978.473541] Lustre: Skipped 1 previous similar message [ 7986.145681] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7986.584602] Lustre: server umount lustre-MDT0000 complete [ 7990.715380] LustreError: 255384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368452 with bad export cookie 16849154072090601199 [ 7990.718187] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7990.736678] LustreError: 255384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7991.112572] Lustre: server umount lustre-MDT0001 complete [ 8002.883464] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8006.012985] Key type lgssc unregistered [ 8006.360859] LNet: 260822:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8006.376246] LNetError: 260822:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8006.393703] LNet: Removed LNI 192.168.204.126@tcp [ 8007.361753] Key type .llcrypt unregistered [ 8007.364086] Key type ._llcrypt unregistered [ 8034.262736] Key type ._llcrypt registered [ 8034.266459] Key type .llcrypt registered [ 8034.395076] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8035.677724] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8035.727289] alg: No test for adler32 (adler32-zlib) [ 8036.777595] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 8037.055023] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8038.759658] Key type lgssc registered [ 8039.901607] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8047.875688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8047.897814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8049.427768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8054.228418] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8059.878333] LustreError: 262245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8059.909192] LustreError: 262245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8063.116405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8063.550148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8067.913342] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8076.591705] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8083.984799] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8092.088575] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8092.498789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8094.604846] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8100.700611] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8110.319619] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8135.653400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8135.669845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8135.684339] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8139.747751] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8139.752668] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8139.753559] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8139.797311] Lustre: Skipped 1 previous similar message [ 8141.628391] Lustre: server umount lustre-OST0000 complete [ 8149.994318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8150.010915] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8150.033963] Lustre: Skipped 1 previous similar message [ 8152.528322] Lustre: server umount lustre-MDT0000 complete [ 8155.105093] LustreError: 262245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8155.123961] LustreError: 262245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8157.029933] LustreError: 262225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368618 with bad export cookie 8244352033161642298 [ 8157.037278] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8157.039613] LustreError: 262225:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8157.272978] Lustre: server umount lustre-MDT0001 complete [ 8172.688944] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8176.013161] Key type lgssc unregistered [ 8176.342500] LNet: 266555:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8176.347835] LNetError: 266555:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8177.386402] LNet: Removed LNI 192.168.204.126@tcp [ 8178.484417] Key type .llcrypt unregistered [ 8178.490361] Key type ._llcrypt unregistered [ 8199.122804] Key type ._llcrypt registered [ 8199.125513] Key type .llcrypt registered [ 8199.278211] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8200.254543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8200.296767] alg: No test for adler32 (adler32-zlib) [ 8201.404289] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 8201.634241] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8203.319653] Key type lgssc registered [ 8205.075690] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8213.915612] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8213.953738] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8215.422057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8220.745382] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8225.783181] LustreError: 267779:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8225.811194] LustreError: 267779:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8230.882028] LustreError: 267778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8232.252792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8232.608101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8237.127620] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8246.085411] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8251.990459] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8258.769368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8259.186487] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8260.217857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8265.225953] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8273.036422] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8288.224867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8288.226688] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8288.239557] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8288.248971] Lustre: Skipped 1 previous similar message [ 8288.257034] Lustre: Skipped 1 previous similar message [ 8293.346914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8293.360596] Lustre: Skipped 1 previous similar message [ 8293.958234] Lustre: server umount lustre-OST0000 complete [ 8298.464982] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8298.487506] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8298.493096] Lustre: Skipped 1 previous similar message [ 8303.588273] LustreError: 268506:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8303.600513] LustreError: 268506:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8303.671607] Lustre: server umount lustre-MDT0000 complete [ 8306.692586] LustreError: 267758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368768 with bad export cookie 12525873521787028011 [ 8306.695788] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8306.705338] LustreError: 267758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8307.071366] Lustre: server umount lustre-MDT0001 complete [ 8316.982495] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8319.420824] Key type lgssc unregistered [ 8319.692383] LNet: 271459:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8319.705205] LNetError: 271459:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8319.723318] LNet: Removed LNI 192.168.204.126@tcp [ 8320.619886] Key type .llcrypt unregistered [ 8320.622039] Key type ._llcrypt unregistered [ 8344.897699] Key type ._llcrypt registered [ 8344.899804] Key type .llcrypt registered [ 8345.118510] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8347.135543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8347.151140] alg: No test for adler32 (adler32-zlib) [ 8348.280500] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 8348.566504] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8350.319265] Key type lgssc registered [ 8351.802880] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8360.332831] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8360.349458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8361.770865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8365.830983] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8371.170252] LustreError: 272879:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8371.187756] LustreError: 272879:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8376.304856] LustreError: 272878:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8376.575957] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8377.232229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8382.856788] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8391.204139] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8397.485358] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8404.169968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8404.586177] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8406.654226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8410.649071] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8419.280297] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8447.969624] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8447.998251] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8448.002525] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8448.021698] Lustre: Skipped 1 previous similar message [ 8452.505691] Lustre: server umount lustre-OST0000 complete [ 8458.208837] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8458.221275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8458.226800] Lustre: Skipped 2 previous similar messages [ 8458.251323] Lustre: Skipped 1 previous similar message [ 8462.834796] Lustre: server umount lustre-MDT0000 complete [ 8463.328927] LustreError: 273822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8463.361760] LustreError: 273822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8466.938298] LustreError: 272860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787368928 with bad export cookie 5266800591348880792 [ 8466.951534] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8467.367959] Lustre: server umount lustre-MDT0001 complete [ 8479.027990] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8482.133663] Key type lgssc unregistered [ 8482.456556] LNet: 277194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8482.478381] LNetError: 277194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8482.498254] LNet: Removed LNI 192.168.204.126@tcp [ 8483.504360] Key type .llcrypt unregistered [ 8483.506703] Key type ._llcrypt unregistered [ 8501.344901] Key type ._llcrypt registered [ 8501.347353] Key type .llcrypt registered [ 8501.456709] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8502.827218] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8502.905045] alg: No test for adler32 (adler32-zlib) [ 8503.996937] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 8504.273870] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8505.969909] Key type lgssc registered [ 8506.982857] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8514.410917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8514.446722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8516.019693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8520.045947] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8525.281362] LustreError: 278433:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8525.292239] LustreError: 278433:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8526.983711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8527.301839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8531.136851] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8538.050833] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8542.942032] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8549.429668] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8549.771639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8555.208720] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8561.085755] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8561.750547] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8573.415984] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8573.424829] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8573.441136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8575.570753] Lustre: server umount lustre-OST0000 complete [ 8581.606855] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8581.607811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8581.617498] Lustre: Skipped 1 previous similar message [ 8586.721891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8586.731901] Lustre: Skipped 2 previous similar messages [ 8591.841512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8591.844680] Lustre: Skipped 1 previous similar message [ 8594.399954] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8596.638086] Lustre: server umount lustre-MDT0000 complete [ 8596.965971] LustreError: 278415:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8596.995246] LustreError: 278415:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8600.682918] LustreError: 278398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369062 with bad export cookie 6137000196312448083 [ 8600.692959] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8600.696086] LustreError: 278398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8601.179844] Lustre: server umount lustre-MDT0001 complete [ 8613.308151] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8616.403865] Key type lgssc unregistered [ 8616.716061] LNet: 282136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8616.721081] LNetError: 282136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8616.740048] LNet: Removed LNI 192.168.204.126@tcp [ 8617.896160] Key type .llcrypt unregistered [ 8617.897981] Key type ._llcrypt unregistered [ 8630.076111] Key type ._llcrypt registered [ 8630.078027] Key type .llcrypt registered [ 8630.161448] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8646.874389] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 8662.120106] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8662.943098] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8663.105941] alg: No test for adler32 (adler32-zlib) [ 8664.251694] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 8664.584839] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8666.263192] Key type lgssc registered [ 8667.414759] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8704.722172] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8704.749235] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8705.973876] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8706.006519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8706.085333] Lustre: lustre-MDT0000: new disk, initializing [ 8706.187329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8706.208421] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8710.878291] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8722.089822] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8722.204773] Lustre: 285688:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8722.260110] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8722.263596] Lustre: Skipped 1 previous similar message [ 8722.323232] Lustre: lustre-MDT0001: new disk, initializing [ 8722.424782] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8722.494794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8722.505430] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8727.685288] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8736.921663] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8741.945256] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8748.247321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8748.653700] Lustre: lustre-OST0000: new disk, initializing [ 8748.656959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8748.673586] Lustre: 287658:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8748.805804] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8755.662192] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8757.810566] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8757.822765] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8757.938504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8766.017782] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8771.204671] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8771.365890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8775.971674] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8776.209513] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8778.726362] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8778.727948] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8778.744304] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8778.750180] LustreError: Skipped 1 previous similar message [ 8778.798288] Lustre: Skipped 1 previous similar message [ 8783.841745] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8783.858208] Lustre: Skipped 2 previous similar messages [ 8784.631905] Lustre: server umount lustre-OST0000 complete [ 8788.968792] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8788.991780] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8788.992330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8789.001778] Lustre: Skipped 1 previous similar message [ 8794.082265] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8794.092785] Lustre: Skipped 2 previous similar messages [ 8799.202511] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8799.205732] Lustre: Skipped 1 previous similar message [ 8803.296357] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8803.543254] Lustre: server umount lustre-MDT0000 complete [ 8804.332518] LustreError: 287944:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8804.345619] LustreError: 287944:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8807.371282] LustreError: 286485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369269 with bad export cookie 1123526729426109570 [ 8807.384635] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8807.385060] LustreError: 286485:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8807.743244] Lustre: server umount lustre-MDT0001 complete [ 8816.413574] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 23:27:55 (1787369275) [ 8823.274798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8823.859323] LustreError: 290076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8823.943089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8829.286278] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8829.411479] LustreError: 290077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8833.513408] LustreError: 290076:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8838.617544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8838.634632] LustreError: 290077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8839.086504] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8844.576476] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8852.753635] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8858.206404] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8866.039854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8866.608139] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8873.991153] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 8882.869582] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8894.436157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8894.439035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8894.443393] LustreError: Skipped 1 previous similar message [ 8894.455910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8894.458569] Lustre: Skipped 1 previous similar message [ 8894.473073] Lustre: Skipped 1 previous similar message [ 8899.392905] Lustre: server umount lustre-OST0000 complete [ 8904.678487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8904.703027] Lustre: Skipped 1 previous similar message [ 8909.794036] LustreError: 290077:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8909.930871] Lustre: server umount lustre-MDT0000 complete [ 8913.949821] LustreError: 290056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369375 with bad export cookie 1123526729426110452 [ 8913.977030] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8914.438471] Lustre: server umount lustre-MDT0001 complete [ 8925.547909] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 8928.759804] Key type lgssc unregistered [ 8929.088571] LNet: 293615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8929.095131] LNetError: 293615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8929.128681] LNet: Removed LNI 192.168.204.126@tcp [ 8930.183591] Key type .llcrypt unregistered [ 8930.185535] Key type ._llcrypt unregistered [ 8958.750992] Key type ._llcrypt registered [ 8958.753311] Key type .llcrypt registered [ 8958.838782] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 8974.100707] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 8975.378309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8975.391860] alg: No test for adler32 (adler32-zlib) [ 8976.483698] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 8976.721903] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 8978.359231] Key type lgssc registered [ 8980.244333] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9030.211305] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9030.244530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9031.651606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9031.722846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9031.817723] Lustre: lustre-MDT0000: new disk, initializing [ 9031.928654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9031.955193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9036.572897] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9048.312292] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9048.401790] Lustre: 297440:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9048.445842] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9048.451388] Lustre: Skipped 1 previous similar message [ 9048.517251] Lustre: lustre-MDT0001: new disk, initializing [ 9048.627383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9048.667593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9048.683083] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9052.982585] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9062.370556] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9068.512411] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9076.174423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9076.462467] Lustre: lustre-OST2710: new disk, initializing [ 9076.465702] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9076.471670] Lustre: 299409:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9076.542554] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9082.242490] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9086.011898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9086.027039] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9086.248272] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9092.376929] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9100.831630] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9101.033388] Lustre: lustre-OST03e8: new disk, initializing [ 9101.043221] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9101.052821] Lustre: 300475:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9101.165687] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9107.602521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9107.610494] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9107.782542] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9109.222855] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9117.087822] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9125.164794] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9125.550887] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9131.167299] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9131.420669] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9136.968249] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9137.258666] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9141.816226] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9142.156294] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9148.904801] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9148.919065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9148.936424] Lustre: Skipped 1 previous similar message [ 9148.969393] Lustre: Skipped 3 previous similar messages [ 9149.488101] Lustre: server umount lustre-MDT0000 complete [ 9154.114687] LustreError: 298235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369615 with bad export cookie 9759192769531631380 [ 9154.115431] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9154.133143] LustreError: 298235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9154.431222] Lustre: server umount lustre-MDT0001 complete [ 9169.801354] Lustre: server umount lustre-OST2710 complete [ 9184.705560] Lustre: server umount lustre-OST03e8 complete [ 9207.153207] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [ 9218.716192] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9260.703338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9260.973512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9261.009386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9261.096897] Lustre: lustre-MDT0000: new disk, initializing [ 9261.194208] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9261.245778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9265.211758] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9276.538816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9276.664171] Lustre: 305729:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9276.721088] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9276.725018] Lustre: Skipped 1 previous similar message [ 9276.801506] Lustre: lustre-MDT0001: new disk, initializing [ 9276.915390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9276.954995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9276.971045] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9282.899040] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9292.988579] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9299.669920] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9307.168405] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9307.375279] Lustre: lustre-OST0000: new disk, initializing [ 9307.379652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9307.385838] Lustre: 307698:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9307.470454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9313.871987] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9314.967308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9314.990449] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9315.228799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9325.461805] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9330.932976] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9331.201212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9336.169978] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9336.502519] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9340.912179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9340.923977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9340.951828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9340.964032] Lustre: Skipped 3 previous similar messages [ 9346.021794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9346.032502] Lustre: Skipped 2 previous similar messages [ 9351.144920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9351.152124] Lustre: Skipped 1 previous similar message [ 9353.185422] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9353.469388] Lustre: server umount lustre-OST0000 complete [ 9358.816766] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9358.827765] LustreError: Skipped 1 previous similar message [ 9358.837084] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9358.855493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9361.393382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9371.617476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9371.630186] Lustre: Skipped 5 previous similar messages [ 9372.127451] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9372.413575] Lustre: server umount lustre-MDT0000 complete [ 9376.642950] LustreError: 305719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369838 with bad export cookie 9759192769531633060 [ 9376.644278] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9376.659255] LustreError: 305719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9377.199378] Lustre: server umount lustre-MDT0001 complete [ 9387.250450] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 23:37:25 (1787369845) [ 9389.181089] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9391.151273] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 23:37:30 (1787369850) [ 9395.652630] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9427.723400] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9428.189190] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9428.230201] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9428.461622] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9433.449479] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9442.659565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9442.892133] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9443.003502] Lustre: 311556:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9448.307564] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9470.903594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9471.099463] Lustre: Denying initial registration attempt from nid 192.168.204.126@tcp, specified as failover [ 9471.117836] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9471.128359] LustreError: 313012:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 9471.133445] LustreError: 313012:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9471.138196] LustreError: 313012:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9471.191667] Lustre: server umount lustre-OST0000 complete [ 9471.197442] LustreError: 313012:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9479.648343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9479.656321] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9479.672183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9484.838474] Lustre: server umount lustre-MDT0000 complete [ 9485.281072] LustreError: 313014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9485.299170] LustreError: 313014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9489.260355] LustreError: 311550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787369951 with bad export cookie 9759192769531633942 [ 9489.271692] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9489.273442] LustreError: 311550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9489.646318] Lustre: server umount lustre-MDT0001 complete [ 9502.641793] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9505.991718] Key type lgssc unregistered [ 9506.323613] LNet: 314136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9506.354392] LNetError: 314136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9506.380105] LNet: Removed LNI 192.168.204.126@tcp [ 9507.361470] Key type .llcrypt unregistered [ 9507.365974] Key type ._llcrypt unregistered [ 9519.586253] Key type ._llcrypt registered [ 9519.592930] Key type .llcrypt registered [ 9519.767107] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9521.300155] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9521.338565] alg: No test for adler32 (adler32-zlib) [ 9522.420599] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 9522.743272] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9524.463144] Key type lgssc registered [ 9525.719589] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9578.667402] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9589.124493] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9589.154798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9590.583510] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9590.632952] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9590.799972] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9596.238680] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9606.565855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9606.720406] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9606.753344] Lustre: 316904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9607.242884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9612.723661] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9621.460794] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9627.916657] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9648.580226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9648.960540] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9649.289593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9657.310445] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9666.337902] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9669.099412] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9669.105110] Lustre: Skipped 1 previous similar message [ 9669.105699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9669.111702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9674.211466] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9674.220266] Lustre: Skipped 2 previous similar messages [ 9675.336617] Lustre: server umount lustre-OST0000 complete [ 9683.936718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9683.957088] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9683.980023] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9685.933827] Lustre: server umount lustre-MDT0000 complete [ 9690.227539] LustreError: 316895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370152 with bad export cookie 2211921793125594775 [ 9690.229598] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9690.234963] LustreError: 316895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9690.604688] Lustre: server umount lustre-MDT0001 complete [ 9703.652622] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9707.178641] Key type lgssc unregistered [ 9707.521468] LNet: 320482:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9707.540015] LNetError: 320482:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9707.565386] LNet: Removed LNI 192.168.204.126@tcp [ 9708.851377] Key type .llcrypt unregistered [ 9708.858322] Key type ._llcrypt unregistered [ 9736.810660] Key type ._llcrypt registered [ 9736.812616] Key type .llcrypt registered [ 9736.993605] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9738.299700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9738.501298] alg: No test for adler32 (adler32-zlib) [ 9739.631901] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 9739.872388] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9741.551129] Key type lgssc registered [ 9743.027508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9751.990923] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9752.036556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9753.488974] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9759.165931] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9763.824606] LustreError: 321901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9763.847301] LustreError: 321901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9768.906148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9768.935120] LustreError: 321900:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9769.496292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9775.246607] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9784.183161] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9790.528481] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9799.742428] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9800.356651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9808.698603] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9819.018645] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9830.760565] Lustre: server umount lustre-MDT0000 complete [ 9830.880921] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9830.890215] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9830.921155] LustreError: 321901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9834.465753] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9834.466369] LustreError: 322628:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9834.514394] LustreError: 322628:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9835.613742] LustreError: 321879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370297 with bad export cookie 9089849541556501259 [ 9835.619139] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9835.625871] LustreError: 321879:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9839.586887] LustreError: 321896:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9839.591800] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9839.628096] LustreError: 321896:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9839.634701] Lustre: Skipped 1 previous similar message [ 9839.646400] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9840.184246] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [ 9842.002433] Lustre: server umount lustre-MDT0001 complete [ 9849.773716] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9860.676986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9861.092885] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x7e25a1ff7a3916e8 [ 9861.104170] Lustre: MGC192.168.204.126@tcp: Connection restored to 0@lo (at 0@lo) [ 9861.611136] LustreError: 323732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9861.641910] LustreError: 323732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9861.758596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9861.887508] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9867.911219] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9870.954283] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9877.155630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9877.812190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9881.910394] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [ 9883.113046] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9890.290509] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9891.636622] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9897.274209] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9898.798364] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9907.680977] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9907.689065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9907.712060] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9907.726157] Lustre: Skipped 1 previous similar message [ 9910.315629] Lustre: server umount lustre-OST0000 complete [ 9918.945741] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9918.975354] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9918.985552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9918.985561] Lustre: Skipped 2 previous similar messages [ 9918.993930] Lustre: Skipped 2 previous similar messages [ 9922.916679] Lustre: server umount lustre-MDT0000 complete [ 9924.064067] LustreError: 325498:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9924.120705] LustreError: 325498:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9929.304416] LustreError: 321881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370391 with bad export cookie 9089849541556508392 [ 9929.311878] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9929.321712] LustreError: 321881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9930.061922] Lustre: server umount lustre-MDT0001 complete [ 9941.186657] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [ 9944.388315] Key type lgssc unregistered [ 9944.670216] LNet: 328172:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9944.675066] LNetError: 328172:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9944.704893] LNet: Removed LNI 192.168.204.126@tcp [ 9946.097836] Key type .llcrypt unregistered [ 9946.099862] Key type ._llcrypt unregistered [ 9976.917486] Key type ._llcrypt registered [ 9976.919826] Key type .llcrypt registered [ 9977.087127] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [ 9978.582496] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9978.758139] alg: No test for adler32 (adler32-zlib) [ 9979.946870] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [ 9980.228117] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [ 9982.023279] Key type lgssc registered [ 9983.464448] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9993.246588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9993.269583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9995.063782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9999.759738] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10004.459315] LustreError: 329589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10004.480590] LustreError: 329589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10009.023123] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10009.330954] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10014.517559] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10021.842917] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10027.602402] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10034.784891] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10034.805789] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10034.813594] Lustre: Skipped 1 previous similar message [10037.195645] Lustre: server umount lustre-MDT0000 complete [10040.289607] LustreError: 329589:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10040.302728] LustreError: 329589:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10041.088578] LustreError: 329570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370502 with bad export cookie 9641922411463371512 [10041.092287] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10041.101427] LustreError: 329570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10041.798888] Lustre: server umount lustre-MDT0001 complete [10060.313924] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10060.740486] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10060.779525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10060.974522] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10065.802283] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10075.059546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10075.213249] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10075.263525] Lustre: 332179:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10075.619709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10080.705941] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10089.216743] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10095.667245] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10105.521863] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10105.775571] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10105.789633] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10105.799402] LustreError: 334062:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [10105.807823] LustreError: 334062:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [10105.819698] LustreError: 334062:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10105.936958] Lustre: server umount lustre-OST0000 complete [10105.939906] LustreError: 334062:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10117.043986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10117.403603] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10117.622453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10124.814097] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10125.451667] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10132.367597] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10139.975339] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10140.106678] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10140.199392] Lustre: lustre-OST0001: new disk, initializing [10140.206475] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10140.220771] Lustre: 335608:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10140.354991] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10146.118244] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10146.924876] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10146.935357] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10147.004417] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10152.809317] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10157.539825] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10157.561667] Lustre: Skipped 2 previous similar messages [10157.562202] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10157.566267] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10157.592131] Lustre: Skipped 1 previous similar message [10161.902524] Lustre: server umount lustre-OST0001 complete [10162.663259] LustreError: 334529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10162.687699] LustreError: 334529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10166.243584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10166.252258] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10166.271726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10167.791703] LustreError: 334529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10167.827435] LustreError: 334529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10171.827244] Lustre: server umount lustre-OST0000 complete [10178.019501] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10178.022596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10178.041778] Lustre: Skipped 1 previous similar message [10178.044882] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10178.053252] LustreError: Skipped 1 previous similar message [10178.071885] Lustre: Skipped 4 previous similar messages [10181.787964] Lustre: server umount lustre-MDT0000 complete [10183.140054] LustreError: 332976:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10183.160519] LustreError: 332976:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10185.380577] LustreError: 335607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370647 with bad export cookie 9641922411463372065 [10185.386569] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10185.392813] LustreError: 335607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10185.732376] Lustre: server umount lustre-MDT0001 complete [10196.620137] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [10199.657121] Key type lgssc unregistered [10200.029929] LNet: 337568:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10200.044967] LNetError: 337568:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10200.073438] LNet: Removed LNI 192.168.204.126@tcp [10201.053869] Key type .llcrypt unregistered [10201.060738] Key type ._llcrypt unregistered [10274.220524] Key type ._llcrypt registered [10274.223657] Key type .llcrypt registered [10274.401991] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [10293.079982] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10294.680266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10294.956582] alg: No test for adler32 (adler32-zlib) [10296.418922] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [10296.612709] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [10298.247323] Key type lgssc registered [10299.936912] Lustre: Echo OBD driver; http://www.lustre.org/ [10337.103669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10337.128558] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10338.300722] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10338.333646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10338.443411] Lustre: lustre-MDT0000: new disk, initializing [10338.520563] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10338.569855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10344.267177] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10356.050244] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10356.229516] Lustre: 342506:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10356.304297] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10356.312682] Lustre: Skipped 1 previous similar message [10356.466532] Lustre: lustre-MDT0001: new disk, initializing [10356.679702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10356.748357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10356.771415] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10362.047353] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10371.334754] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10376.271709] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10383.092765] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10383.400699] Lustre: lustre-OST0000: new disk, initializing [10383.404256] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10383.409398] Lustre: 344472:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10383.489318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10389.404496] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10391.112788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10391.129400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10391.321324] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10400.060646] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10406.144923] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10406.434815] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10411.294225] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10411.610264] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10417.124493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10417.140828] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10417.160716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10417.173208] Lustre: Skipped 1 previous similar message [10422.245212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10422.258029] Lustre: Skipped 1 previous similar message [10427.366881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10427.370618] Lustre: Skipped 1 previous similar message [10427.875754] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10428.081489] Lustre: server umount lustre-OST0000 complete [10432.483985] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10432.487793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10432.492190] Lustre: Skipped 2 previous similar messages [10432.498390] Lustre: Skipped 1 previous similar message [10437.601389] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10437.617594] Lustre: Skipped 1 previous similar message [10446.303146] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10446.711865] Lustre: server umount lustre-MDT0000 complete [10450.606245] LustreError: 342499:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787370912 with bad export cookie 17499834872052710901 [10450.619054] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10451.141198] Lustre: server umount lustre-MDT0001 complete [10460.022623] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 23:55:19 (1787370919) [10468.121915] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 23:55:26 (1787370926) [10476.871051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10477.306712] LustreError: 347230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10477.322374] LustreError: 347230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10477.366281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10482.123818] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10482.663405] LustreError: 347231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10487.777541] LustreError: 347230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10490.886167] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10491.393211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10496.248673] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10505.112402] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10510.690258] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10517.265092] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10517.541742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10524.344287] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10531.990293] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10543.611537] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10545.515868] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10547.739322] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10550.491547] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10552.524206] Lustre: Failing over lustre-MDT0000 [10552.829069] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10552.836629] LustreError: Skipped 1 previous similar message [10552.843864] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10552.871436] LustreError: 348429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.906402] LustreError: 348429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10552.953211] Lustre: server umount lustre-MDT0000 complete [10557.410029] LustreError: 347230:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10557.415076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10557.437204] LustreError: 347230:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10557.481082] Lustre: Skipped 1 previous similar message [10567.653937] LustreError: 347227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10567.670895] LustreError: 347227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [10572.586475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10572.706647] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10572.768077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (not set up) [10572.778502] Lustre: Skipped 2 previous similar messages [10573.032623] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10573.094930] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10574.365483] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10578.165738] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10578.439708] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10578.512431] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10578.625079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10588.433427] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10590.219602] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10594.556427] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10596.623841] Lustre: Failing over lustre-MDT0000 [10597.029468] Lustre: server umount lustre-MDT0000 complete [10598.882987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10598.894184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10598.903229] LustreError: 347227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10598.912343] Lustre: Skipped 2 previous similar messages [10598.945423] LustreError: 347227:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10614.305330] Lustre: 340730:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787371060/real 1787371060] req@000000005af7ada4 x1874194080034304/t0(0) o400->MGC192.168.204.126@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787371076 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10614.345941] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10616.883841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10624.497353] LustreError: 340726:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000007f87d2e x1874194080040064/t0(0) o250->MGC192.168.204.126@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10624.968880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10625.067523] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10625.563260] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10629.896919] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10630.128748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10630.132147] Lustre: Skipped 2 previous similar messages [10630.161401] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [10630.223766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10642.326651] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10644.890503] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10655.715029] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10655.728651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10655.731282] Lustre: Skipped 1 previous similar message [10655.737416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10655.738505] Lustre: Skipped 3 previous similar messages [10658.189553] Lustre: server umount lustre-MDT0000 complete [10660.832613] LustreError: 348429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10660.870816] LustreError: 348429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 26 previous similar messages [10662.329589] LustreError: 351212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371124 with bad export cookie 17499834872052713610 [10662.345909] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10662.350195] LustreError: 351212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10662.730348] Lustre: server umount lustre-MDT0001 complete [10677.223988] Lustre: server umount lustre-OST0000 complete [10695.814305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10696.368827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10701.177195] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10711.210350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10716.324530] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10724.061232] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10729.638893] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10736.922487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10737.127304] LustreError: 355407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10737.146976] LustreError: 355407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10741.364220] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10744.730307] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10752.894674] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10760.843370] Lustre: server umount lustre-OST0000 complete [10767.335382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10767.337207] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10767.357331] Lustre: Skipped 1 previous similar message [10767.367384] Lustre: Skipped 1 previous similar message [10775.729710] Lustre: server umount lustre-MDT0000 complete [10778.591269] Lustre: 340728:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787371224/real 1787371224] req@000000004a111963 x1874194080085504/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787371240 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10779.936935] LustreError: 353570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371241 with bad export cookie 17499834872052714359 [10779.943338] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10779.952851] LustreError: 353570:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10780.322755] Lustre: server umount lustre-MDT0001 complete [10792.422615] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [10795.746621] Key type lgssc unregistered [10796.020805] LNet: 357125:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10796.032033] LNetError: 357125:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10796.048907] LNet: Removed LNI 192.168.204.126@tcp [10797.064464] Key type .llcrypt unregistered [10797.066600] Key type ._llcrypt unregistered [10826.099836] Key type ._llcrypt registered [10826.102755] Key type .llcrypt registered [10826.222331] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [10827.752366] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10827.779859] alg: No test for adler32 (adler32-zlib) [10828.785268] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [10829.099221] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [10830.815182] Key type lgssc registered [10832.022299] Lustre: Echo OBD driver; http://www.lustre.org/ [10840.373511] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10840.403159] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10841.888948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10846.608497] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10851.296805] LustreError: 358667:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.316663] LustreError: 358667:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10856.274580] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10856.418083] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10856.552325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10861.560659] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10869.043536] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10874.986469] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10881.904954] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10882.300863] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10886.453831] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10888.851424] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10895.860025] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10912.231666] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10912.239901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10912.262021] Lustre: Skipped 1 previous similar message [10915.168879] Lustre: server umount lustre-MDT0000 complete [10917.347055] LustreError: 359672:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10917.373477] LustreError: 359672:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10919.323298] LustreError: 358649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371381 with bad export cookie 8261313445618185212 [10919.325608] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10919.336911] LustreError: 358649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10919.962265] Lustre: server umount lustre-MDT0001 complete [10934.954918] Lustre: server umount lustre-OST0000 complete [10952.535483] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10953.292208] LustreError: 362401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10953.450820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10958.840972] LustreError: 362402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10959.339330] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10963.941918] LustreError: 362401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10970.163595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10970.468660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10976.915849] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [10985.174918] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10992.045991] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10999.604322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10999.996604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11006.570892] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11015.804638] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11024.569060] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11040.225330] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11040.239411] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11040.251136] Lustre: Skipped 3 previous similar messages [11040.266689] Lustre: Skipped 3 previous similar messages [11045.350141] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11045.363791] Lustre: Skipped 3 previous similar messages [11045.535314] Lustre: server umount lustre-MDT0000 complete [11049.435748] LustreError: 362382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371511 with bad export cookie 8261313445618186325 [11049.441320] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11049.447857] LustreError: 362382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11049.908341] Lustre: server umount lustre-MDT0001 complete [11065.110169] Lustre: server umount lustre-OST0000 complete [11085.757268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11086.878730] LustreError: 366338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11086.889571] LustreError: 366338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11086.952944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11096.516241] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11106.820211] LustreError: 366338:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11106.842896] LustreError: 366338:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11109.845774] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11114.908923] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11123.443157] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11129.548885] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11136.634416] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11136.996583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11137.001858] Lustre: Skipped 1 previous similar message [11143.964304] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11151.515519] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11156.484672] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11164.640235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11164.655962] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11164.667027] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11166.701817] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11166.706920] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11168.690660] Lustre: server umount lustre-OST0000 complete [11176.929967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11176.936189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11176.952485] Lustre: Skipped 1 previous similar message [11176.966192] Lustre: Skipped 3 previous similar messages [11187.167294] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11187.359143] Lustre: server umount lustre-MDT0000 complete [11191.079455] LustreError: 366319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371652 with bad export cookie 8261313445618187347 [11191.088760] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11191.095749] LustreError: 366319:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11191.500850] Lustre: server umount lustre-MDT0001 complete [11202.838706] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11205.910808] Key type lgssc unregistered [11206.222151] LNet: 369878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11206.233014] LNetError: 369878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11206.252993] LNet: Removed LNI 192.168.204.126@tcp [11207.209972] Key type .llcrypt unregistered [11207.212048] Key type ._llcrypt unregistered [11234.969605] Key type ._llcrypt registered [11234.972196] Key type .llcrypt registered [11235.092254] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11236.277982] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11236.392133] alg: No test for adler32 (adler32-zlib) [11237.426298] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [11237.698358] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11239.360084] Key type lgssc registered [11241.184160] Lustre: Echo OBD driver; http://www.lustre.org/ [11249.687978] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11249.721264] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11249.730628] LustreError: 371318:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11249.754974] LustreError: 371318:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11249.766528] LustreError: 371318:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11249.778279] LustreError: 371318:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11249.788098] LustreError: 371318:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11258.121898] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11258.133054] LustreError: 371716:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11258.147679] LustreError: 371716:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11258.152726] LustreError: 371716:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11258.157313] LustreError: 371716:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11258.161598] LustreError: 371716:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11277.327714] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11280.030674] Key type lgssc unregistered [11280.293660] LNet: 372724:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11280.305294] LNetError: 372724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11280.330077] LNet: Removed LNI 192.168.204.126@tcp [11280.967868] Key type .llcrypt unregistered [11280.970509] Key type ._llcrypt unregistered [11301.670760] Key type ._llcrypt registered [11301.673656] Key type .llcrypt registered [11301.788498] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [11317.005135] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11318.649592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11318.697151] alg: No test for adler32 (adler32-zlib) [11319.844934] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [11320.109052] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11321.831419] Key type lgssc registered [11323.237221] Lustre: Echo OBD driver; http://www.lustre.org/ [11357.355314] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11357.377905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11358.640582] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11358.705445] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11358.857723] Lustre: lustre-MDT0000: new disk, initializing [11359.028504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11359.047123] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11363.323692] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11373.714035] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11373.806613] Lustre: 375885:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11373.877503] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11373.884277] Lustre: Skipped 1 previous similar message [11373.980550] Lustre: lustre-MDT0001: new disk, initializing [11374.089776] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11374.143970] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11374.162299] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11378.571430] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11388.131043] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11395.291273] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11401.790629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11402.049722] Lustre: lustre-OST0000: new disk, initializing [11402.053729] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11402.059465] Lustre: 377850:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11402.177364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11411.159643] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11411.600680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11411.613878] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11411.729730] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11420.658839] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11425.773456] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11426.106190] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11431.781712] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11432.012609] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11437.541747] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11437.544120] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11437.550097] LustreError: Skipped 1 previous similar message [11437.550694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11437.575760] Lustre: Skipped 1 previous similar message [11442.664184] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11442.669182] Lustre: Skipped 2 previous similar messages [11447.783477] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11447.792898] Lustre: Skipped 1 previous similar message [11448.287549] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11448.560336] Lustre: server umount lustre-OST0000 complete [11452.913045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11452.924944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11452.930051] Lustre: Skipped 1 previous similar message [11458.016727] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11458.025569] Lustre: Skipped 2 previous similar messages [11458.930573] Lustre: server umount lustre-MDT0000 complete [11462.908448] LustreError: 376678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787371924 with bad export cookie 5962244977421869502 [11462.912819] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11462.918983] LustreError: 376678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11463.203779] Lustre: server umount lustre-MDT0001 complete [11472.056591] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11473.762876] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 00:12:12 (1787371932) [11481.104041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11481.797207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11486.862797] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11487.201756] LustreError: 380313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11487.225691] LustreError: 380313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11491.385884] LustreError: 380312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11495.908502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11496.316800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11501.307722] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11508.736847] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11514.859562] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11521.770076] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11522.060483] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11528.296158] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11535.842676] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11543.454812] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11543.669909] Lustre: lustre-OST0001: new disk, initializing [11543.682688] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11543.706969] Lustre: 383148:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11543.812408] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11549.814301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11549.828677] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11549.942826] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11551.106899] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11562.442748] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11569.220666] Lustre: lustre-OST0001: Not available for connect from 192.168.204.26@tcp (stopping) [11570.663129] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11570.677026] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11570.689475] Lustre: Skipped 1 previous similar message [11570.707418] LustreError: Skipped 1 previous similar message [11572.608631] Lustre: server umount lustre-OST0001 complete [11574.316710] LustreError: 382135:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11580.897997] LustreError: 382134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11580.928415] LustreError: 382134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11586.020393] LustreError: 382134:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11586.043342] LustreError: 382134:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11591.139856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11591.151592] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11591.166571] Lustre: Skipped 1 previous similar message [11591.170944] Lustre: Skipped 2 previous similar messages [11596.307251] Lustre: server umount lustre-OST0000 complete [11601.376460] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11601.392513] Lustre: Skipped 1 previous similar message [11606.232539] Lustre: server umount lustre-MDT0000 complete [11606.501709] LustreError: 380313:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11606.515032] LustreError: 380313:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11610.175510] LustreError: 380293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372072 with bad export cookie 5962244977421870377 [11610.178779] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11610.190591] LustreError: 380293:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11616.839394] Lustre: server umount lustre-MDT0001 complete [11628.390879] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [11631.357706] Key type lgssc unregistered [11631.677650] LNet: 385139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11631.683663] LNetError: 385139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11631.704300] LNet: Removed LNI 192.168.204.126@tcp [11632.593281] Key type .llcrypt unregistered [11632.595693] Key type ._llcrypt unregistered [11685.119675] Key type ._llcrypt registered [11685.123162] Key type .llcrypt registered [11685.234895] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [11686.691756] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11686.747015] alg: No test for adler32 (adler32-zlib) [11687.878734] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [11688.165452] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [11689.943178] Key type lgssc registered [11691.242706] Lustre: Echo OBD driver; http://www.lustre.org/ [11698.528254] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11698.547472] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11699.847318] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11699.876942] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11700.044846] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11705.086303] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11713.557914] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11713.635213] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11713.663318] Lustre: 387940:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11713.881077] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11719.400514] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11727.296618] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11733.369941] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11739.617260] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11739.629954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11739.657058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11740.136767] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11740.163215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11740.175917] Lustre: Skipped 1 previous similar message [11741.801040] LustreError: 389532:0:(ldlm_resource.c:1207:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000d31f37b8) refcount nonzero (2) after lock cleanup; forcing cleanup. [11741.884216] Lustre: server umount lustre-MDT0000 complete [11746.001920] LustreError: 387930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372207 with bad export cookie 16036477498719835284 [11746.003461] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11746.016024] LustreError: 387930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11746.302558] Lustre: server umount lustre-MDT0001 complete [11757.630792] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 00:16:56 (1787372216) [11765.990840] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11766.523809] LustreError: 390451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11766.540392] LustreError: 390451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11766.594564] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11771.218759] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11771.878192] LustreError: 390452:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11776.999490] LustreError: 390451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11779.757723] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11780.145388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11785.034293] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11792.273327] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11798.020824] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11804.658652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11804.899613] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11805.086036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11811.291053] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11818.210443] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11826.918767] Lustre: Permanently deactivating lustre-OST0000 [11826.940826] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11835.964168] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11838.630812] LustreError: 393487:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11838.648080] LustreError: 393487:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11838.660218] LustreError: 393487:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11842.212565] LustreError: 393536:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11842.221197] LustreError: 393536:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11850.442804] Lustre: server umount lustre-OST0000 complete [11853.260453] LustreError: 393786:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11853.266553] LustreError: 393786:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11856.354108] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11856.357725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11856.374833] Lustre: Skipped 1 previous similar message [11856.389686] Lustre: Skipped 1 previous similar message [11861.472793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11861.477649] Lustre: Skipped 1 previous similar message [11864.640437] Lustre: server umount lustre-MDT0000 complete [11866.595957] LustreError: 391436:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11866.635608] LustreError: 391436:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11868.288780] LustreError: 390434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372330 with bad export cookie 16036477498719835865 [11868.295911] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11868.307936] LustreError: 390434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11879.080716] Lustre: server umount lustre-MDT0001 complete [11888.677097] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11893.962923] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11896.979521] LustreError: 394946:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11903.452840] Lustre: 395285:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.126@tcp [11905.627221] Lustre: server umount MGS complete [11917.652741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11918.070082] LustreError: 396008:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11918.099157] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11918.102877] Lustre: Skipped 1 previous similar message [11918.151154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11923.149908] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11928.554578] LustreError: 396009:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11928.583148] LustreError: 396009:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11931.862274] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11932.130976] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11937.109319] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [11945.583131] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11953.012852] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11955.032500] Lustre: Permanently reactivating lustre-OST0000 [11955.056285] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11955.064951] Lustre: Skipped 2 previous similar messages [11981.294133] LustreError: 397743:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11987.999223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11988.373776] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11988.385531] Lustre: Skipped 1 previous similar message [11994.710802] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12003.062596] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12010.977594] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12010.988157] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12011.010843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12012.005830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12022.244574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12022.251100] Lustre: Skipped 4 previous similar messages [12023.264866] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12023.518369] Lustre: server umount lustre-OST0000 complete [12027.367927] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12027.385272] Lustre: Skipped 1 previous similar message [12033.554890] Lustre: server umount lustre-MDT0000 complete [12037.497674] LustreError: 396742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372499 with bad export cookie 16036477498719836915 [12037.507867] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12037.518072] LustreError: 396742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12037.909445] Lustre: server umount lustre-MDT0001 complete [12047.901371] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12050.623250] Key type lgssc unregistered [12051.014198] LNet: 400328:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12051.025261] LNetError: 400328:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12051.054123] LNet: Removed LNI 192.168.204.126@tcp [12052.022954] Key type .llcrypt unregistered [12052.026951] Key type ._llcrypt unregistered [12068.935599] Key type ._llcrypt registered [12068.943436] Key type .llcrypt registered [12069.101385] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [12083.132936] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12084.053372] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12084.164917] alg: No test for adler32 (adler32-zlib) [12085.170245] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [12085.351351] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12087.025406] Key type lgssc registered [12087.940849] Lustre: Echo OBD driver; http://www.lustre.org/ [12127.143906] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12127.163789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12128.578959] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12128.651209] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12128.714337] Lustre: lustre-MDT0000: new disk, initializing [12128.816181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12128.851877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12133.963565] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12144.182522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12144.348518] Lustre: 403486:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12144.469316] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12144.477864] Lustre: Skipped 1 previous similar message [12144.631682] Lustre: lustre-MDT0001: new disk, initializing [12144.751666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12144.801475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12144.814987] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12149.681786] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12159.720973] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12165.445391] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12172.650925] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12172.914523] Lustre: lustre-OST0000: new disk, initializing [12172.920850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12172.926802] Lustre: 405455:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12173.042503] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12179.290886] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12181.609454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12181.625849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12181.663330] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12188.389599] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12192.925820] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12193.137103] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12197.365758] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12197.559114] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12202.478268] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12202.493204] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12202.493486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12202.502627] Lustre: Skipped 1 previous similar message [12202.525831] Lustre: Skipped 1 previous similar message [12205.759458] Lustre: server umount lustre-OST0000 complete [12211.169378] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12211.175514] LustreError: Skipped 1 previous similar message [12211.179173] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12211.187474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12212.705350] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12212.707904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12215.294574] Lustre: server umount lustre-MDT0000 complete [12218.907940] LustreError: 403477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372680 with bad export cookie 11570062436164793132 [12218.917569] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12218.921994] LustreError: 403477:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12219.307639] Lustre: server umount lustre-MDT0001 complete [12228.749710] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12230.710629] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 00:24:49 (1787372689) [12238.851444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12239.330505] LustreError: 407915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12239.352292] LustreError: 407915:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12239.424876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12244.157454] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12244.450893] LustreError: 407916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12249.574544] LustreError: 407915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12253.419823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12253.747047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12258.442794] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12265.395052] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12271.579635] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12278.670612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12279.108759] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12285.800847] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12293.481290] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12295.596199] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12298.058308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12309.987023] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12309.998254] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12310.017673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12310.022210] Lustre: Skipped 1 previous similar message [12314.081417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12314.087255] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12314.092362] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12323.295905] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12323.488456] Lustre: server umount lustre-OST0000 complete [12329.443345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12329.463163] Lustre: Skipped 1 previous similar message [12329.463836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12329.480258] Lustre: Skipped 4 previous similar messages [12333.893916] Lustre: server umount lustre-MDT0000 complete [12334.577188] LustreError: 408918:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12334.609685] LustreError: 408918:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12337.987102] LustreError: 407895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372799 with bad export cookie 11570062436164794014 [12337.998228] LustreError: 407895:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12337.998922] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12338.462443] Lustre: server umount lustre-MDT0001 complete [12350.834903] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12354.018883] Key type lgssc unregistered [12354.425617] LNet: 411649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12354.433104] LNetError: 411649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12354.464034] LNet: Removed LNI 192.168.204.126@tcp [12355.605766] Key type .llcrypt unregistered [12355.609733] Key type ._llcrypt unregistered [12383.980567] Key type ._llcrypt registered [12383.982338] Key type .llcrypt registered [12384.087115] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12384.527351] Key type .llcrypt unregistered [12384.537161] Key type ._llcrypt unregistered [12404.874579] Key type ._llcrypt registered [12404.878330] Key type .llcrypt registered [12405.001605] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12406.181731] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12406.392812] alg: No test for adler32 (adler32-zlib) [12407.450949] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [12407.666259] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12409.335219] Key type lgssc registered [12410.646995] Lustre: Echo OBD driver; http://www.lustre.org/ [12418.913180] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12418.950581] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12420.386899] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12423.706212] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12430.821789] LustreError: 413869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12430.844950] LustreError: 413869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12431.832223] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12432.138644] LustreError: 413868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12432.206401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12437.475071] LustreError: 413869:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12437.509210] LustreError: 413869:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12440.375616] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12442.612058] LustreError: 413862:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12442.627646] LustreError: 413862:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12445.692673] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12449.591982] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12450.794797] LustreError: 413868:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12450.816531] LustreError: 413868:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12456.837636] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12457.266235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12462.077491] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12471.268932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12471.277534] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12471.291779] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12472.811579] LustreError: 414593:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12472.818325] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12472.825807] LustreError: 414593:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12472.834555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12472.925257] Lustre: server umount lustre-OST0000 complete [12477.354242] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12477.365832] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12477.376734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12477.788169] Lustre: server umount lustre-MDT0000 complete [12482.727165] LustreError: 413850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787372944 with bad export cookie 1703493700285234834 [12482.737041] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12482.740432] LustreError: 413850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12483.307935] Lustre: server umount lustre-MDT0001 complete [12496.354522] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12499.378919] Key type lgssc unregistered [12499.736890] LNet: 417019:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12499.748578] LNetError: 417019:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12499.762874] LNet: Removed LNI 192.168.204.126@tcp [12500.664054] Key type .llcrypt unregistered [12500.669336] Key type ._llcrypt unregistered [12527.162844] Key type ._llcrypt registered [12527.169946] Key type .llcrypt registered [12527.308461] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12528.453251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12528.502189] alg: No test for adler32 (adler32-zlib) [12529.608327] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [12529.862234] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12531.543213] Key type lgssc registered [12532.712411] Lustre: Echo OBD driver; http://www.lustre.org/ [12540.977349] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12540.991634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12556.575410] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12561.695975] LustreError: 418409:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.126@tcp: failed processing log, type 4: rc = -110 [12588.384392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12595.900943] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12604.784702] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12611.866058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12612.328890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12612.407225] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12617.161363] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12617.708993] LustreError: 418434:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12617.718662] LustreError: 418434:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12622.823429] LustreError: 418435:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12622.855727] LustreError: 418435:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12626.239100] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12626.599523] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12631.820360] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12639.515555] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12646.344421] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12655.924066] Lustre: server umount lustre-OST0000 complete [12661.741944] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12661.757917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12661.788809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12662.757860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12662.760135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12662.812165] Lustre: Skipped 1 previous similar message [12667.874179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12667.886814] Lustre: Skipped 1 previous similar message [12671.046575] Lustre: server umount lustre-MDT0000 complete [12673.003251] LustreError: 419446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12673.034774] LustreError: 419446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12674.015138] Lustre: 417747:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787373119/real 1787373119] req@000000000c7ea31a x1874196421442176/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787373135 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12674.022285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12674.071677] Lustre: 417747:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [12675.856839] LustreError: 418415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373137 with bad export cookie 16991087788626814335 [12675.877719] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12675.880986] LustreError: 418415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12676.554515] Lustre: server umount lustre-MDT0001 complete [12687.815990] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12690.900903] Key type lgssc unregistered [12691.171710] LNet: 421982:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12691.185125] LNetError: 421982:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12691.203572] LNet: Removed LNI 192.168.204.126@tcp [12692.114931] Key type .llcrypt unregistered [12692.118768] Key type ._llcrypt unregistered [12717.709804] Key type ._llcrypt registered [12717.715322] Key type .llcrypt registered [12717.835171] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12718.913475] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12719.039512] alg: No test for adler32 (adler32-zlib) [12720.118859] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [12720.406297] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12722.119595] Key type lgssc registered [12723.464832] Lustre: Echo OBD driver; http://www.lustre.org/ [12731.215841] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12731.248427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12732.759304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12736.760254] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12743.137530] LustreError: 423401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12743.173761] LustreError: 423401:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12745.665554] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12746.186608] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12750.275564] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12758.061165] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12764.149629] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12771.862953] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12772.378344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12778.494330] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12778.557692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12786.246506] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12792.291797] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12792.303753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12792.341702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12794.443110] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.26@tcp (stopping) [12795.876334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12795.877768] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12795.908395] Lustre: Skipped 1 previous similar message [12798.147038] Lustre: server umount lustre-MDT0000 complete [12799.514592] LustreError: 423397:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12800.603613] LustreError: 423382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.26@tcp arrived at 1787373262 with bad export cookie 12848736244931140702 [12800.993489] LustreError: 423402:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12801.006744] LustreError: 423402:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12802.649335] LustreError: 423384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373264 with bad export cookie 12848736244931139813 [12802.662584] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12802.665550] LustreError: 423384:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12806.112931] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12806.113675] LustreError: 424629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12806.120621] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12806.132313] LustreError: 424629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12806.143894] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12806.151707] Lustre: Skipped 2 previous similar messages [12809.178621] Lustre: server umount lustre-OST0000 complete [12812.256030] LustreError: 424629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12812.272511] LustreError: 424629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12817.069669] Lustre: server umount lustre-MDT0001 complete [12828.407489] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12831.795969] Key type lgssc unregistered [12832.098986] LNet: 427091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12832.109934] LNetError: 427091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12832.143418] LNet: Removed LNI 192.168.204.126@tcp [12833.099080] Key type .llcrypt unregistered [12833.106135] Key type ._llcrypt unregistered [12857.713231] Key type ._llcrypt registered [12857.717984] Key type .llcrypt registered [12857.840309] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [12859.142687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12859.160919] alg: No test for adler32 (adler32-zlib) [12860.245738] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [12860.487274] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [12862.220244] Key type lgssc registered [12863.472576] Lustre: Echo OBD driver; http://www.lustre.org/ [12871.834252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12871.861529] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12873.394614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12878.029611] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12883.942203] LustreError: 428508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12883.958389] LustreError: 428508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12887.023121] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12887.438369] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12892.249531] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12900.442785] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12906.556453] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12914.556932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12915.103359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12918.213514] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12922.995072] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [12930.858827] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12937.707216] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12937.723731] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12937.731233] Lustre: Skipped 1 previous similar message [12938.349686] Lustre: server umount lustre-MDT0001 complete [12938.820916] LustreError: 428502:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.204.26@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12942.818678] LustreError: 428508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12942.835639] LustreError: 428508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12947.936895] LustreError: 428507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12947.940343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12947.955631] Lustre: Skipped 1 previous similar message [12947.960539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12951.266767] Lustre: server umount lustre-OST0000 complete [12953.058776] LustreError: 429510:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12955.255387] Lustre: server umount lustre-MDT0000 complete [12969.487251] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [12972.228050] Key type lgssc unregistered [12972.569833] LNet: 432196:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12972.584874] LNetError: 432196:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12973.610229] LNet: Removed LNI 192.168.204.126@tcp [12974.571336] Key type .llcrypt unregistered [12974.572914] Key type ._llcrypt unregistered [12999.589231] Key type ._llcrypt registered [12999.595942] Key type .llcrypt registered [12999.761496] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [13014.725752] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13015.954340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13016.102028] alg: No test for adler32 (adler32-zlib) [13017.150925] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [13017.422113] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13019.087172] Key type lgssc registered [13020.232271] Lustre: Echo OBD driver; http://www.lustre.org/ [13057.185955] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13072.567881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13072.659602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13073.922737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13073.971922] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13074.064437] Lustre: lustre-MDT0000: new disk, initializing [13074.187841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13074.209387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13080.046181] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13096.584140] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13097.001744] Lustre: lustre-OST0000: new disk, initializing [13097.008121] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13097.020138] Lustre: Skipped 1 previous similar message [13097.038416] Lustre: 437231:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13097.148696] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13104.732843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13104.740159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13104.779638] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13105.007586] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13114.315548] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13130.513275] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13130.622079] Lustre: 436214:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13130.686746] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13130.753809] Lustre: lustre-MDT0001: new disk, initializing [13130.870112] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13130.914693] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13130.935887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13135.539177] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13144.192996] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13144.472402] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13148.662127] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13148.898827] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13154.276572] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13154.279538] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [13154.289837] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13154.306649] Lustre: Skipped 1 previous similar message [13157.444505] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.26@tcp (stopping) [13159.397741] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13159.408209] Lustre: Skipped 1 previous similar message [13159.624943] Lustre: server umount lustre-MDT0001 complete [13169.638066] LustreError: 436225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13169.657679] LustreError: 436225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13174.754587] LustreError: 436221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13174.768964] LustreError: 436221:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13179.877770] LustreError: 436225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13179.879438] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13179.916143] Lustre: Skipped 1 previous similar message [13179.929718] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13185.003701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13185.005074] LustreError: 436224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13190.115124] LustreError: 436225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13191.135167] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13191.543886] Lustre: server umount lustre-OST0000 complete [13201.847950] Lustre: server umount lustre-MDT0000 complete [13214.898747] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13217.675935] Key type lgssc unregistered [13217.921915] LNet: 440986:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13217.927832] LNetError: 440986:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13217.941526] LNet: Removed LNI 192.168.204.126@tcp [13218.854665] Key type .llcrypt unregistered [13218.857317] Key type ._llcrypt unregistered [13263.089802] Key type ._llcrypt registered [13263.092500] Key type .llcrypt registered [13263.223276] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [13263.584266] Key type .llcrypt unregistered [13263.588415] Key type ._llcrypt unregistered [13282.512777] Key type ._llcrypt registered [13282.516981] Key type .llcrypt registered [13282.706512] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [13283.904346] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13284.022245] alg: No test for adler32 (adler32-zlib) [13285.125690] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [13285.428547] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [13287.162948] Key type lgssc registered [13288.419943] Lustre: Echo OBD driver; http://www.lustre.org/ [13317.713451] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13317.787489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13319.103395] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13319.156902] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13319.344618] Lustre: lustre-MDT0000: new disk, initializing [13319.458832] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13319.473411] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13323.831939] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13335.519565] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13335.641760] Lustre: 444374:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13335.704167] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13335.707842] Lustre: Skipped 1 previous similar message [13335.815123] Lustre: lustre-MDT0001: new disk, initializing [13335.930133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13335.975412] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13335.990584] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13340.796873] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13350.759769] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13356.463364] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13364.007918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13364.368274] Lustre: lustre-OST0000: new disk, initializing [13364.376537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13364.394605] Lustre: 446339:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13364.508787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13371.511277] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13371.529307] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13371.678177] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13371.696318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13381.777688] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13390.311669] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13390.317107] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13390.320593] Lustre: Skipped 2 previous similar messages [13390.333389] Lustre: Skipped 2 previous similar messages [13395.425202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13395.441663] Lustre: Skipped 2 previous similar messages [13395.694395] Lustre: server umount lustre-MDT0000 complete [13399.280031] LustreError: 444366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787373861 with bad export cookie 2593185062798440704 [13399.280687] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13399.288662] LustreError: 444366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13399.768642] Lustre: server umount lustre-MDT0001 complete [13414.295295] Lustre: server umount lustre-OST0000 complete [13425.852031] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 00:44:44 (1787373884) [13433.932836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13434.808736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13439.529247] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13439.982555] LustreError: 448371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13440.008171] LustreError: 448371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13445.091145] LustreError: 448372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13450.208649] LustreError: 448371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13450.535810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13451.028831] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13456.772692] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13465.957733] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13472.563621] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13492.773332] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13493.048870] Lustre: Found index 0 for lustre-OST0000, updating log [13493.423940] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13501.871208] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [13505.714191] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13509.196076] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13516.260501] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13516.266865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13516.285876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13516.292432] Lustre: Skipped 1 previous similar message [13520.926897] Lustre: server umount lustre-OST0000 complete [13536.337330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13536.352730] Lustre: Skipped 1 previous similar message [13536.360018] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13536.365711] Lustre: Skipped 3 previous similar messages [13541.344843] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13541.350972] Lustre: Skipped 1 previous similar message [13542.684377] Lustre: server umount lustre-MDT0000 complete [13543.392740] LustreError: 449806:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13543.413643] LustreError: 449806:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13546.405509] LustreError: 450966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374008 with bad export cookie 2593185062798442097 [13546.411705] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13546.427327] LustreError: 450966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13546.888156] Lustre: server umount lustre-MDT0001 complete [13555.563829] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 00:46:54 (1787374014) [13562.079394] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [14179.284561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14179.935395] LustreError: 454784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14179.971979] LustreError: 454784:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14180.104249] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14184.089930] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14185.440201] LustreError: 454785:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14185.460349] LustreError: 454785:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14189.550212] LustreError: 454784:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14193.543364] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14193.922086] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14198.872682] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14207.220373] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14213.704570] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14235.311264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14235.601335] Lustre: Found index 0 for lustre-OST0000, updating log [14236.050838] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14239.968266] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [14243.058404] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14252.415638] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14254.979483] Lustre: server umount lustre-OST0000 complete [14260.709896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14260.711638] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14260.737692] Lustre: Skipped 1 previous similar message [14260.741490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14260.760582] Lustre: Skipped 1 previous similar message [14261.797862] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14261.811111] Lustre: Skipped 1 previous similar message [14263.843729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14263.856470] Lustre: Skipped 3 previous similar messages [14266.108533] Lustre: server umount lustre-MDT0000 complete [14266.922355] LustreError: 455787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14266.953220] LustreError: 455787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14270.270163] LustreError: 454765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787374732 with bad export cookie 2593185062798443182 [14270.276243] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14270.280779] LustreError: 454765:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14270.650906] Lustre: server umount lustre-MDT0001 complete [14299.425223] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp [14303.893618] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [14314.428172] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [14354.722723] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 01:00:13 (1787374813) [14375.569651] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [14379.129674] Key type lgssc unregistered [14379.586663] LNet: 461407:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14379.595305] LNetError: 461407:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14497.383291] LNet: Removed LNI 192.168.204.126@tcp [14498.357318] Key type .llcrypt unregistered [14498.359578] Key type ._llcrypt unregistered [14518.372346] Key type ._llcrypt registered [14518.379230] Key type .llcrypt registered [14518.469785] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [14519.944466] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14520.045911] alg: No test for adler32 (adler32-zlib) [14521.257149] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [14521.581273] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [14523.311194] Key type lgssc registered [14524.651555] Lustre: Echo OBD driver; http://www.lustre.org/ [14529.669344] LNet: Added LNI 192.168.204.126@tcp100 [8/256/0/180] [14530.800459] LNet: Added LNI 192.168.204.126@tcp101 [8/256/0/180] [14531.799887] LNet: Added LNI 192.168.204.126@tcp102 [8/256/0/180] [14532.788264] LNet: Added LNI 192.168.204.126@tcp103 [8/256/0/180] [14533.699867] LNet: Added LNI 192.168.204.126@tcp104 [8/256/0/180] [14534.561344] LNet: Added LNI 192.168.204.126@tcp105 [8/256/0/180] [14535.346592] LNet: Added LNI 192.168.204.126@tcp106 [8/256/0/180] [14536.346274] LNet: Added LNI 192.168.204.126@tcp107 [8/256/0/180] [14537.297736] LNet: Added LNI 192.168.204.126@tcp108 [8/256/0/180] [14538.290369] LNet: Added LNI 192.168.204.126@tcp109 [8/256/0/180] [14539.235112] LNet: Added LNI 192.168.204.126@tcp110 [8/256/0/180] [14540.280487] LNet: Added LNI 192.168.204.126@tcp111 [8/256/0/180] [14541.275931] LNet: Added LNI 192.168.204.126@tcp112 [8/256/0/180] [14542.437926] LNet: Added LNI 192.168.204.126@tcp113 [8/256/0/180] [14543.518935] LNet: Added LNI 192.168.204.126@tcp114 [8/256/0/180] [14544.668858] LNet: Added LNI 192.168.204.126@tcp115 [8/256/0/180] [14545.773356] LNet: Added LNI 192.168.204.126@tcp116 [8/256/0/180] [14546.782426] LNet: Added LNI 192.168.204.126@tcp117 [8/256/0/180] [14547.632264] LNet: Added LNI 192.168.204.126@tcp118 [8/256/0/180] [14548.657737] LNet: Added LNI 192.168.204.126@tcp119 [8/256/0/180] [14549.688225] LNet: Added LNI 192.168.204.126@tcp120 [8/256/0/180] [14550.841972] LNet: Added LNI 192.168.204.126@tcp121 [8/256/0/180] [14551.939200] LNet: Added LNI 192.168.204.126@tcp122 [8/256/0/180] [14553.194968] LNet: Added LNI 192.168.204.126@tcp123 [8/256/0/180] [14554.410389] LNet: Added LNI 192.168.204.126@tcp124 [8/256/0/180] [14555.509279] LNet: Added LNI 192.168.204.126@tcp125 [8/256/0/180] [14556.682384] LNet: Added LNI 192.168.204.126@tcp126 [8/256/0/180] [14557.694120] LNet: Added LNI 192.168.204.126@tcp127 [8/256/0/180] [14559.002701] LNet: Added LNI 192.168.204.126@tcp128 [8/256/0/180] [14560.216590] LNet: Added LNI 192.168.204.126@tcp129 [8/256/0/180] [14561.294470] LNet: Added LNI 192.168.204.126@tcp130 [8/256/0/180] [14562.526767] LNet: Added LNI 192.168.204.126@tcp131 [8/256/0/180] [14563.683510] LNet: Added LNI 192.168.204.126@tcp132 [8/256/0/180] [14564.724455] LNet: Added LNI 192.168.204.126@tcp133 [8/256/0/180] [14565.897512] LNet: Added LNI 192.168.204.126@tcp134 [8/256/0/180] [14566.926691] LNet: Added LNI 192.168.204.126@tcp135 [8/256/0/180] [14574.659825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14574.701952] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14575.870765] Lustre: 464457:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [14575.988723] LNetError: 464483:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [14576.006545] LNetError: Protocol error connecting to 192.168.204.126@tcp118 at host 192.168.204.126:988: is it running a compatible version of Lustre? [14576.033840] LNetError: Skipped 11 previous similar messages [14576.164446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14576.226564] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14576.322174] Lustre: lustre-MDT0000: new disk, initializing [14576.406888] Lustre: 464514:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [14576.420722] Lustre: 464514:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14576.442557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14576.462253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14576.928169] LNet: 7 local NIs in recovery (showing 7): 192.168.204.126@tcp124, 192.168.204.126@tcp126, 192.168.204.126@tcp128, 192.168.204.126@tcp129, 192.168.204.126@tcp127, 192.168.204.126@tcp125, 192.168.204.126@tcp118 [14578.019802] LNetError: 461917:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [14578.030465] LNetError: 461917:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [14578.037585] LNetError: Protocol error connecting to 192.168.204.126@tcp126 at host 192.168.204.126:988: is it running a compatible version of Lustre? [14578.045740] LNetError: Skipped 2 previous similar messages [14580.069753] LNetError: 461915:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [14580.082763] LNetError: 461915:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [14580.088579] LNetError: Protocol error connecting to 192.168.204.126@tcp126 at host 192.168.204.126:988: is it running a compatible version of Lustre? [14581.393435] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14584.162438] LNetError: 464483:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [14584.178963] LNetError: Protocol error connecting to 192.168.204.126@tcp126 at host 192.168.204.126:988: is it running a compatible version of Lustre? [14592.320201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14592.358388] LNetError: 464489:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [14592.367398] LNetError: Protocol error connecting to 192.168.204.126@tcp126 at host 192.168.204.126:988: is it running a compatible version of Lustre? [14592.526570] Lustre: 464472:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14592.612597] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14592.616209] Lustre: Skipped 1 previous similar message [14592.701859] Lustre: lustre-MDT0001: new disk, initializing [14592.751339] Lustre: 465278:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [14592.762470] Lustre: 465278:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14592.889968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14592.929053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14592.943126] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14598.272965] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14607.585841] LNetError: 464487:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [14607.595260] LNetError: Protocol error connecting to 192.168.204.126@tcp123 at host 192.168.204.126:988: is it running a compatible version of Lustre? [14609.129728] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14615.689651] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14622.989573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14623.409710] Lustre: lustre-OST0000: new disk, initializing [14623.415383] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14623.422508] Lustre: 466450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14623.577606] Lustre: 466455:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [14623.594573] Lustre: 466455:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [14623.680722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14623.968415] LNetError: 464488:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [14623.983902] LNetError: 464488:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 13 previous similar messages [14624.002450] LNetError: Protocol error connecting to 192.168.204.126@tcp121 at host 192.168.204.126:988: is it running a compatible version of Lustre? [14624.016448] LNetError: Skipped 13 previous similar messages [14631.207810] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14631.979594] Lustre: 466835:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.126@tcp: can't add NID 192.168.204.126@tcp130: rc = -75 [14631.988844] Lustre: 466835:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14632.077301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14632.090480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14632.240358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14636.450364] LNet: 1 local NIs in recovery (showing 1): 192.168.204.126@tcp126 [14641.558956] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14652.902484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14652.902937] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14652.912723] LustreError: Skipped 1 previous similar message [14652.924080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14652.939080] Lustre: Skipped 1 previous similar message [14657.185172] Lustre: server umount lustre-OST0000 complete [14657.769792] LNetError: 464489:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.126 [14657.798179] LNetError: 464489:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 28 previous similar messages [14657.808747] LNetError: Protocol error connecting to 192.168.204.126@tcp120 at host 192.168.204.126:988: is it running a compatible version of Lustre? [14657.827933] LNetError: Skipped 28 previous similar messages [14663.141305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14663.154145] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14663.162748] Lustre: Skipped 2 previous similar messages [14663.190321] Lustre: Skipped 1 previous similar message [14668.250994] Lustre: server umount lustre-MDT0000 complete [14672.752949] LustreError: 465277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375134 with bad export cookie 16661937357473330574 [14672.772293] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14672.795695] LustreError: 465277:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14673.168495] Lustre: server umount lustre-MDT0001 complete [14685.502418] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [14689.000698] Key type lgssc unregistered [14689.443682] LNet: 468366:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14689.452723] LNetError: 468366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14689.458216] LNet: Removed LNI 192.168.204.126@tcp [14690.463176] LNet: Removed LNI 192.168.204.126@tcp100 [14690.465286] LNetError: 468366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14690.487798] LNetError: 468366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14690.498844] LNet: Removed LNI 192.168.204.126@tcp101 [14691.551331] LNet: Removed LNI 192.168.204.126@tcp102 [14691.553274] LNetError: 468366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14691.560135] LNetError: 468366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14691.572561] LNet: Removed LNI 192.168.204.126@tcp103 [14692.639178] LNet: Removed LNI 192.168.204.126@tcp104 [14693.663285] LNet: Removed LNI 192.168.204.126@tcp105 [14693.672344] LNetError: 468366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14693.699459] LNetError: 468366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14693.711726] LNet: Removed LNI 192.168.204.126@tcp106 [14694.752281] LNet: Removed LNI 192.168.204.126@tcp107 [14695.777283] LNet: Removed LNI 192.168.204.126@tcp108 [14696.799190] LNet: Removed LNI 192.168.204.126@tcp109 [14697.824043] LNet: Removed LNI 192.168.204.126@tcp110 [14697.835260] LNetError: 468366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14697.872360] LNetError: 468366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14697.891549] LNet: Removed LNI 192.168.204.126@tcp111 [14698.911270] LNet: Removed LNI 192.168.204.126@tcp112 [14699.935168] LNet: Removed LNI 192.168.204.126@tcp113 [14700.959465] LNet: Removed LNI 192.168.204.126@tcp114 [14701.984056] LNet: Removed LNI 192.168.204.126@tcp115 [14703.008178] LNet: Removed LNI 192.168.204.126@tcp116 [14704.031321] LNet: Removed LNI 192.168.204.126@tcp117 [14705.055173] LNet: Removed LNI 192.168.204.126@tcp118 [14706.079860] LNet: Removed LNI 192.168.204.126@tcp119 [14706.083626] LNetError: 468366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14706.090313] LNetError: 468366:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14707.103184] LNet: Removed LNI 192.168.204.126@tcp120 [14708.127454] LNet: Removed LNI 192.168.204.126@tcp121 [14709.151414] LNet: Removed LNI 192.168.204.126@tcp122 [14710.175196] LNet: Removed LNI 192.168.204.126@tcp123 [14711.199155] LNet: Removed LNI 192.168.204.126@tcp124 [14712.223298] LNet: Removed LNI 192.168.204.126@tcp125 [14712.231134] LNet: Removed LNI 192.168.204.126@tcp126 [14713.247159] LNet: Removed LNI 192.168.204.126@tcp127 [14714.273724] LNet: Removed LNI 192.168.204.126@tcp128 [14715.295130] LNet: Removed LNI 192.168.204.126@tcp129 [14715.300807] LNet: Removed LNI 192.168.204.126@tcp130 [14715.305804] LNet: Removed LNI 192.168.204.126@tcp131 [14715.310273] LNet: Removed LNI 192.168.204.126@tcp132 [14715.314172] LNet: Removed LNI 192.168.204.126@tcp133 [14715.316827] LNet: Removed LNI 192.168.204.126@tcp134 [14715.329060] LNet: Removed LNI 192.168.204.126@tcp135 [14716.131926] Key type .llcrypt unregistered [14716.134796] Key type ._llcrypt unregistered [14748.292551] Key type ._llcrypt registered [14748.296600] Key type .llcrypt registered [14748.514769] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [14766.001027] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [14767.437816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14767.635909] alg: No test for adler32 (adler32-zlib) [14768.961510] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [14769.345344] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [14771.064151] Key type lgssc registered [14772.330264] Lustre: Echo OBD driver; http://www.lustre.org/ [14813.878609] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 01:07:52 (1787375272) [14835.016157] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [14838.677174] Key type lgssc unregistered [14838.941409] LNet: 472462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14838.945700] LNetError: 472462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14838.971456] LNet: Removed LNI 192.168.204.126@tcp [14839.966545] Key type .llcrypt unregistered [14839.971592] Key type ._llcrypt unregistered [14862.405144] Key type ._llcrypt registered [14862.407578] Key type .llcrypt registered [14862.579696] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [14864.675106] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14864.746749] alg: No test for adler32 (adler32-zlib) [14865.750386] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [14866.016891] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [14867.759959] Key type lgssc registered [14869.329193] Lustre: Echo OBD driver; http://www.lustre.org/ [14881.266064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14881.334654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14883.003618] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14883.054383] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14883.247091] Lustre: lustre-MDT0000: new disk, initializing [14883.422003] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14883.492371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14890.075678] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14905.292860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14905.460979] Lustre: 473676:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14905.520271] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14905.523458] Lustre: Skipped 1 previous similar message [14905.572479] Lustre: lustre-MDT0001: new disk, initializing [14905.640818] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14905.669043] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14905.684730] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14911.614892] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14923.486756] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14930.351463] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14938.923099] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14939.398774] Lustre: lustre-OST0000: new disk, initializing [14939.403519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14939.412298] Lustre: 475647:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14939.515822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14947.108712] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [14948.039628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14948.059945] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14948.229862] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14958.585715] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14963.232816] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing lnet_if_list [14969.833543] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14969.875387] LNet: Added LNI 192.168.204.126@tcp10 [8/256/0/180] [14976.989171] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14977.047741] LNet: Added LNI 192.168.204.126@tcp11 [8/256/0/180] [14982.082415] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14982.120453] LNet: Added LNI 192.168.204.126@tcp12 [8/256/0/180] [14986.785265] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14986.812727] LNet: Added LNI 192.168.204.126@tcp13 [8/256/0/180] [14992.101091] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14992.180556] LNet: Added LNI 192.168.204.126@tcp14 [8/256/0/180] [14998.215354] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14998.251142] LNet: Added LNI 192.168.204.126@tcp15 [8/256/0/180] [15004.554793] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15004.594391] LNet: Added LNI 192.168.204.126@tcp16 [8/256/0/180] [15011.004929] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15011.044905] LNet: Added LNI 192.168.204.126@tcp17 [8/256/0/180] [15018.060591] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15018.100089] LNet: Added LNI 192.168.204.126@tcp18 [8/256/0/180] [15024.274868] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15024.304090] LNet: Added LNI 192.168.204.126@tcp19 [8/256/0/180] [15030.365353] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15030.387967] LNet: Added LNI 192.168.204.126@tcp20 [8/256/0/180] [15036.787557] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15036.847094] LNet: Added LNI 192.168.204.126@tcp21 [8/256/0/180] [15042.802864] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15042.826793] LNet: Added LNI 192.168.204.126@tcp22 [8/256/0/180] [15048.861197] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15048.886240] LNet: Added LNI 192.168.204.126@tcp23 [8/256/0/180] [15057.465453] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15057.519418] LNet: Added LNI 192.168.204.126@tcp24 [8/256/0/180] [15065.398836] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15065.471638] LNet: Added LNI 192.168.204.126@tcp25 [8/256/0/180] [15072.684811] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15072.738040] LNet: Added LNI 192.168.204.126@tcp26 [8/256/0/180] [15080.044342] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15080.072627] LNet: Added LNI 192.168.204.126@tcp27 [8/256/0/180] [15086.484848] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15086.507663] LNet: Added LNI 192.168.204.126@tcp28 [8/256/0/180] [15092.872168] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15092.937743] LNet: Added LNI 192.168.204.126@tcp29 [8/256/0/180] [15100.183221] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15100.235331] LNet: Added LNI 192.168.204.126@tcp30 [8/256/0/180] [15108.234496] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15108.262673] LNet: Added LNI 192.168.204.126@tcp31 [8/256/0/180] [15115.725654] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15115.761503] LNet: Added LNI 192.168.204.126@tcp32 [8/256/0/180] [15123.498829] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15123.569034] LNet: Added LNI 192.168.204.126@tcp33 [8/256/0/180] [15130.434828] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15130.498889] LNet: Added LNI 192.168.204.126@tcp34 [8/256/0/180] [15138.925478] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15138.986499] LNet: Added LNI 192.168.204.126@tcp35 [8/256/0/180] [15147.761834] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15147.806794] LNet: Added LNI 192.168.204.126@tcp36 [8/256/0/180] [15156.407984] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15156.450247] LNet: Added LNI 192.168.204.126@tcp37 [8/256/0/180] [15164.897099] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15164.922407] LNet: Added LNI 192.168.204.126@tcp38 [8/256/0/180] [15173.201621] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15173.243818] LNet: Added LNI 192.168.204.126@tcp39 [8/256/0/180] [15182.188830] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15182.233655] LNet: Added LNI 192.168.204.126@tcp40 [8/256/0/180] [15190.082129] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15190.112372] LNet: Added LNI 192.168.204.126@tcp41 [8/256/0/180] [15199.176890] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15199.245239] LNet: Added LNI 192.168.204.126@tcp42 [8/256/0/180] [15208.080387] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15208.135450] LNet: Added LNI 192.168.204.126@tcp43 [8/256/0/180] [15216.748396] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15216.793512] LNet: Added LNI 192.168.204.126@tcp44 [8/256/0/180] [15226.701480] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15226.748733] LNet: Added LNI 192.168.204.126@tcp45 [8/256/0/180] [15235.893770] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15235.923946] LNet: Added LNI 192.168.204.126@tcp46 [8/256/0/180] [15245.365747] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15245.402339] LNet: Added LNI 192.168.204.126@tcp47 [8/256/0/180] [15256.628290] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15256.697561] LNet: Added LNI 192.168.204.126@tcp48 [8/256/0/180] [15266.386915] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15266.457803] LNet: Added LNI 192.168.204.126@tcp49 [8/256/0/180] [15278.608458] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15278.626743] LNet: Added LNI 192.168.204.126@tcp50 [8/256/0/180] [15300.065312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15300.072060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15300.085114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15301.601623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15301.607661] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15305.725378] Lustre: server umount lustre-MDT0000 complete [15310.756214] LustreError: 473668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375772 with bad export cookie 14921861740869744233 [15310.756730] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15310.809468] LustreError: 473668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15311.560232] Lustre: server umount lustre-MDT0001 complete [15327.199303] Lustre: 472947:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787375773/real 1787375773] req@000000002b524f7a x1874198871075584/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787375789 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15327.246554] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15327.259733] Lustre: Skipped 1 previous similar message [15331.310902] Lustre: 472948:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787375777/real 1787375777] req@000000006852797a x1874198871075712/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787375793 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15332.390882] Lustre: 472948:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787375778/real 1787375778] req@00000000f78c015a x1874198871075840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787375794 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15332.837336] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15336.415761] Lustre: 472946:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787375782/real 1787375782] req@0000000011421992 x1874198871076096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787375798 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15342.159585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15342.559172] Lustre: 472947:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787375788/real 1787375788] req@000000002fb0fb12 x1874198871076224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787375804 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15356.185575] LustreError: 475641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15356.198578] LustreError: 475641:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [15356.242484] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15357.280335] LustreError: 475642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15357.281938] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15360.373119] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15362.529908] LustreError: 475641:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15367.651106] LustreError: 475642:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15369.509108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15370.006891] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15373.109279] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15375.431766] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15383.986752] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15390.788666] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15411.168880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15411.179330] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15411.205074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15411.226278] Lustre: Skipped 2 previous similar messages [15412.707679] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15414.255334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15414.269978] Lustre: Skipped 2 previous similar messages [15419.374744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15424.991605] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15425.278809] Lustre: server umount lustre-OST0000 complete [15431.649000] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15431.672805] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15431.707822] Lustre: Skipped 1 previous similar message [15431.720431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15431.734692] Lustre: Skipped 3 previous similar messages [15437.537216] Lustre: server umount lustre-MDT0000 complete [15439.840386] LustreError: 499894:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15439.865964] LustreError: 499894:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15441.491116] LustreError: 474471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787375903 with bad export cookie 14921861740869746907 [15441.494445] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15441.510165] LustreError: 474471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15441.844308] Lustre: server umount lustre-MDT0001 complete [15454.368367] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [15456.795491] Key type lgssc unregistered [15457.346799] LNet: 502702:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15457.361213] LNetError: 502702:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15457.375677] LNet: Removed LNI 192.168.204.126@tcp [15457.382164] LNet: Removed LNI 192.168.204.126@tcp10 [15457.389327] LNet: Removed LNI 192.168.204.126@tcp11 [15457.402340] LNet: Removed LNI 192.168.204.126@tcp12 [15457.405422] LNet: Removed LNI 192.168.204.126@tcp13 [15457.415748] LNet: Removed LNI 192.168.204.126@tcp14 [15457.424455] LNet: Removed LNI 192.168.204.126@tcp15 [15457.431771] LNet: Removed LNI 192.168.204.126@tcp16 [15457.434561] LNet: Removed LNI 192.168.204.126@tcp17 [15457.445177] LNet: Removed LNI 192.168.204.126@tcp18 [15457.449766] LNet: Removed LNI 192.168.204.126@tcp19 [15457.456269] LNet: Removed LNI 192.168.204.126@tcp20 [15457.457749] LNet: Removed LNI 192.168.204.126@tcp21 [15457.459216] LNet: Removed LNI 192.168.204.126@tcp22 [15457.461635] LNet: Removed LNI 192.168.204.126@tcp23 [15457.463157] LNet: Removed LNI 192.168.204.126@tcp24 [15457.465345] LNet: Removed LNI 192.168.204.126@tcp25 [15457.467560] LNet: Removed LNI 192.168.204.126@tcp26 [15457.469790] LNet: Removed LNI 192.168.204.126@tcp27 [15457.471873] LNet: Removed LNI 192.168.204.126@tcp28 [15457.473835] LNet: Removed LNI 192.168.204.126@tcp29 [15457.475798] LNet: Removed LNI 192.168.204.126@tcp30 [15457.478148] LNet: Removed LNI 192.168.204.126@tcp31 [15457.479515] LNet: Removed LNI 192.168.204.126@tcp32 [15457.480786] LNet: Removed LNI 192.168.204.126@tcp33 [15457.483023] LNet: Removed LNI 192.168.204.126@tcp34 [15457.485900] LNet: Removed LNI 192.168.204.126@tcp35 [15457.488648] LNet: Removed LNI 192.168.204.126@tcp36 [15457.492460] LNet: Removed LNI 192.168.204.126@tcp37 [15457.496599] LNet: Removed LNI 192.168.204.126@tcp38 [15457.500197] LNet: Removed LNI 192.168.204.126@tcp39 [15457.504439] LNet: Removed LNI 192.168.204.126@tcp40 [15457.507579] LNet: Removed LNI 192.168.204.126@tcp41 [15457.510894] LNet: Removed LNI 192.168.204.126@tcp42 [15457.513737] LNet: Removed LNI 192.168.204.126@tcp43 [15457.517300] LNet: Removed LNI 192.168.204.126@tcp44 [15457.520830] LNet: Removed LNI 192.168.204.126@tcp45 [15457.523670] LNet: Removed LNI 192.168.204.126@tcp46 [15457.527512] LNet: Removed LNI 192.168.204.126@tcp47 [15457.530183] LNet: Removed LNI 192.168.204.126@tcp48 [15457.532533] LNet: Removed LNI 192.168.204.126@tcp49 [15457.543478] LNet: Removed LNI 192.168.204.126@tcp50 [15458.311434] Key type .llcrypt unregistered [15458.313444] Key type ._llcrypt unregistered [15481.135405] Key type ._llcrypt registered [15481.139262] Key type .llcrypt registered [15481.275548] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [15498.510695] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15500.133217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15500.192706] alg: No test for adler32 (adler32-zlib) [15501.323750] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [15501.594050] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [15503.247178] Key type lgssc registered [15504.323659] Lustre: Echo OBD driver; http://www.lustre.org/ [15543.399697] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15543.451668] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15544.975278] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15545.072602] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15545.225935] Lustre: lustre-MDT0000: new disk, initializing [15545.428105] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15545.482855] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15551.656321] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15564.671532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15564.791277] Lustre: 505900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15564.831974] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15564.834159] Lustre: Skipped 1 previous similar message [15564.879872] Lustre: lustre-MDT0001: new disk, initializing [15564.960353] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15565.041824] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15565.058367] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15570.281341] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15581.140371] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15587.675811] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15596.292829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15596.528894] Lustre: *** cfs_fail_loc=90f, val=0*** [15596.612055] Lustre: lustre-OST0000: new disk, initializing [15596.616979] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15596.625440] Lustre: 507919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15596.746041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15602.290222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15602.327864] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15602.499126] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15604.459583] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15614.224675] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15623.136600] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15623.137548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15623.147104] LustreError: Skipped 1 previous similar message [15623.151281] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15623.169821] Lustre: Skipped 1 previous similar message [15623.191256] Lustre: Skipped 1 previous similar message [15628.178521] Lustre: server umount lustre-OST0000 complete [15633.387502] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15633.418715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15638.617392] Lustre: server umount lustre-MDT0000 complete [15642.587309] LustreError: 506694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376104 with bad export cookie 17744263985105474684 [15642.592586] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15642.608970] LustreError: 506694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15643.216278] Lustre: server umount lustre-MDT0001 complete [15655.354450] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [15658.861637] Key type lgssc unregistered [15659.216804] LNet: 509827:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15659.232948] LNetError: 509827:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15659.255814] LNet: Removed LNI 192.168.204.126@tcp [15660.425115] Key type .llcrypt unregistered [15660.428300] Key type ._llcrypt unregistered [15682.837442] Key type ._llcrypt registered [15682.843511] Key type .llcrypt registered [15683.008208] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15684.748105] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15684.784579] alg: No test for adler32 (adler32-zlib) [15686.025387] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [15686.335441] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [15688.031219] Key type lgssc registered [15689.617528] Lustre: Echo OBD driver; http://www.lustre.org/ [15698.231508] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15698.276960] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15699.654829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15704.173359] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15709.162496] LustreError: 511098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15709.203390] LustreError: 511098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15713.640684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15714.072706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15719.057381] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15727.282385] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15733.120396] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15739.358045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15739.737692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15747.175165] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15755.658551] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15768.753929] loop0: detected capacity change from 0 to 204800000 [15768.961367] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15768.975662] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15768.985677] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15774.618725] loop0: detected capacity change from 0 to 204800000 [15774.821324] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15774.838761] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15774.847762] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15780.832817] loop0: detected capacity change from 0 to 204800000 [15787.332804] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15787.497302] Lustre: fs15246-OST0000: new disk, initializing [15787.501570] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15787.514493] Lustre: 514696:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15787.584869] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15794.221465] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15804.269620] loop1: detected capacity change from 0 to 204800000 [15809.589764] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15809.757595] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15809.860437] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15809.930843] Lustre: fs15246-MDT0000: new disk, initializing [15810.092519] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15810.111143] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15810.234462] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15811.303160] Lustre: 515821:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000005312f4b7 x1874199731814784/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787376289 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15814.531551] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [15820.394731] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15820.409639] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15820.456777] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15839.725152] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15839.753365] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15842.109728] Lustre: server umount fs15246-MDT0000 complete [15844.838766] LustreError: 511095:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15849.965363] LustreError: 511094:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15851.518279] Lustre: server umount fs15246-OST0000 complete [15865.314930] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15865.326890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15865.333741] Lustre: Skipped 1 previous similar message [15870.476936] Lustre: server umount lustre-OST0000 complete [15875.555158] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15875.571987] Lustre: Skipped 1 previous similar message [15875.579949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15875.585822] Lustre: Skipped 1 previous similar message [15880.396418] Lustre: server umount lustre-MDT0000 complete [15880.673102] LustreError: 511827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15880.700077] LustreError: 511827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15883.863505] LustreError: 511080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376345 with bad export cookie 10226926129272447382 [15883.876165] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15884.204134] Lustre: server umount lustre-MDT0001 complete [15895.730802] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [15898.585803] Key type lgssc unregistered [15898.887856] LNet: 518919:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15898.894645] LNetError: 518919:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15898.918094] LNet: Removed LNI 192.168.204.126@tcp [15899.725943] Key type .llcrypt unregistered [15899.728086] Key type ._llcrypt unregistered [15918.595169] Key type ._llcrypt registered [15918.597502] Key type .llcrypt registered [15918.685148] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [15919.607427] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15919.648070] alg: No test for adler32 (adler32-zlib) [15920.728722] Lustre: Lustre: Build Version: 2.17.57_45_g4630995 [15920.945964] LNet: Added LNI 192.168.204.126@tcp [8/256/0/180] [15922.639198] Key type lgssc registered [15923.885573] Lustre: Echo OBD driver; http://www.lustre.org/ [15971.502929] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [15985.677979] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [16022.928403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16022.970711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16024.368974] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16024.429380] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16024.584460] Lustre: lustre-MDT0000: new disk, initializing [16024.695924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16024.713694] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16029.609499] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16043.167117] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16043.397425] Lustre: 523389:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16043.495732] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16043.503707] Lustre: Skipped 1 previous similar message [16043.585644] Lustre: lustre-MDT0001: new disk, initializing [16043.744950] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16043.820790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16043.845301] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16049.524503] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16058.473697] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16064.332390] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16070.314918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16070.597672] Lustre: lustre-OST0000: new disk, initializing [16070.605065] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16070.614971] Lustre: 525361:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16070.724532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16077.190950] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_default_debug -1 all [16080.481274] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16080.506901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16080.632607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16086.534535] Lustre: DEBUG MARKER: oleg426-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16091.647081] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16091.864497] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16096.790371] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16096.986585] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16101.347975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16101.356171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16101.363145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16101.374380] Lustre: Skipped 1 previous similar message [16106.468251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16106.474219] Lustre: Skipped 2 previous similar messages [16111.587585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16111.594804] Lustre: Skipped 1 previous similar message [16113.119187] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16113.305040] Lustre: server umount lustre-OST0000 complete [16116.713553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16116.716185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16116.733868] Lustre: Skipped 1 previous similar message [16116.749675] Lustre: Skipped 1 previous similar message [16121.826227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16121.833112] Lustre: Skipped 1 previous similar message [16123.031189] Lustre: server umount lustre-MDT0000 complete [16126.459134] LustreError: 523381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787376588 with bad export cookie 17462148382722492398 [16126.461301] LustreError: MGC192.168.204.126@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16126.467496] LustreError: 523381:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16126.751397] Lustre: server umount lustre-MDT0001 complete [16151.204802] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing set_hostid [16164.304250] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing load_modules_local [16197.478993] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15927 sec ========= 01:30:56 (1787376656) [16199.475721] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 01:30:58 (1787376658) === [16203.586313] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 01:31:02 (1787376662) === [16247.472534] Lustre: DEBUG MARKER: oleg426-server.virtnet: executing unload_modules_local [16250.234760] Key type lgssc unregistered [16250.667910] LNet: 532007:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16250.682789] LNetError: 532007:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16250.720888] LNet: Removed LNI 192.168.204.126@tcp [16251.791834] Key type .llcrypt unregistered [16251.800964] Key type ._llcrypt unregistered