[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 520865928 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.003040] x2apic enabled [ 0.004008] Switched APIC routing to physical x2apic. [ 0.005011] kvm-guest: setup PV IPIs [ 0.007878] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008021] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009015] pid_max: default: 32768 minimum: 301 [ 0.010136] LSM: Security Framework initializing [ 0.011055] Yama: becoming mindful. [ 0.012038] SELinux: Initializing. [ 0.013074] *** VALIDATE selinux *** [ 0.021726] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026185] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028024] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030121] *** VALIDATE tmpfs *** [ 0.032033] *** VALIDATE proc *** [ 0.033189] *** VALIDATE cgroup *** [ 0.034006] *** VALIDATE cgroup2 *** [ 0.035271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036172] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040017] Spectre V2 : User space: Vulnerable [ 0.041009] Speculative Store Bypass: Vulnerable [ 0.044330] debug: unmapping init [mem 0xffffffff9e459000-0xffffffff9e460fff] [ 0.046270] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047864] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048023] ... version: 2 [ 0.049012] ... bit width: 48 [ 0.050014] ... generic registers: 4 [ 0.051013] ... value mask: 0000ffffffffffff [ 0.052013] ... max period: 00007fffffffffff [ 0.053014] ... fixed-purpose events: 3 [ 0.054010] ... event mask: 000000070000000f [ 0.055312] rcu: Hierarchical SRCU implementation. [ 0.057622] smp: Bringing up secondary CPUs ... [ 0.058670] x86: Booting SMP configuration: [ 0.059031] .... node #0, CPUs: #1 #2 #3 [ 0.062373] smp: Brought up 1 node, 4 CPUs [ 0.064011] smpboot: Max logical packages: 1 [ 0.065018] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.136749] node 0 deferred pages initialised in 68ms [ 0.139113] devtmpfs: initialized [ 0.140257] x86/mm: Memory block size: 128MB [ 0.143860] gcov: version magic: 0x41383552 [ 0.146317] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.147064] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.148276] pinctrl core: initialized pinctrl subsystem [ 0.149301] [ 0.149895] ************************************************************* [ 0.150015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.151015] ** ** [ 0.152014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.153018] ** ** [ 0.154017] ** This means that this kernel is built to expose internal ** [ 0.155016] ** IOMMU data structures, which may compromise security on ** [ 0.156014] ** your system. ** [ 0.157016] ** ** [ 0.158014] ** If you see this message and you are not debugging the ** [ 0.159014] ** kernel, report this immediately to your vendor! ** [ 0.160014] ** ** [ 0.161013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.162012] ************************************************************* [ 0.163778] NET: Registered protocol family 16 [ 0.164442] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.165058] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.166066] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.168060] cpuidle: using governor menu [ 0.169773] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.172551] PCI: Using configuration type 1 for base access [ 0.174116] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.183102] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.184037] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.186068] cryptd: max_cpu_qlen set to 1000 [ 0.189498] ACPI: Added _OSI(Module Device) [ 0.191012] ACPI: Added _OSI(Processor Device) [ 0.193014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.194012] ACPI: Added _OSI(Processor Aggregator Device) [ 0.199406] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.206306] ACPI: Interpreter enabled [ 0.209062] ACPI: PM: (supports S0 S3 S4 S5) [ 0.210013] ACPI: Using IOAPIC for interrupt routing [ 0.211133] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.212431] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.222363] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.223048] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.224020] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.225093] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.227479] acpiphp: Slot [2] registered [ 0.228141] acpiphp: Slot [5] registered [ 0.229121] acpiphp: Slot [6] registered [ 0.230157] acpiphp: Slot [7] registered [ 0.232401] acpiphp: Slot [8] registered [ 0.236140] acpiphp: Slot [9] registered [ 0.238262] acpiphp: Slot [10] registered [ 0.240211] acpiphp: Slot [3] registered [ 0.241164] acpiphp: Slot [4] registered [ 0.243227] acpiphp: Slot [11] registered [ 0.245117] acpiphp: Slot [12] registered [ 0.246091] acpiphp: Slot [13] registered [ 0.248104] acpiphp: Slot [14] registered [ 0.250106] acpiphp: Slot [15] registered [ 0.251115] acpiphp: Slot [16] registered [ 0.253179] acpiphp: Slot [17] registered [ 0.255102] acpiphp: Slot [18] registered [ 0.257110] acpiphp: Slot [19] registered [ 0.258113] acpiphp: Slot [20] registered [ 0.260112] acpiphp: Slot [21] registered [ 0.263192] acpiphp: Slot [22] registered [ 0.265218] acpiphp: Slot [23] registered [ 0.266155] acpiphp: Slot [24] registered [ 0.268168] acpiphp: Slot [25] registered [ 0.270149] acpiphp: Slot [26] registered [ 0.271153] acpiphp: Slot [27] registered [ 0.273192] acpiphp: Slot [28] registered [ 0.275181] acpiphp: Slot [29] registered [ 0.276164] acpiphp: Slot [30] registered [ 0.278225] acpiphp: Slot [31] registered [ 0.280130] PCI host bridge to bus 0000:00 [ 0.281031] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.284051] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.286039] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.289027] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.292027] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.295027] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.297212] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.300190] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.304344] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.313716] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.319102] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.322029] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.324022] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.327021] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.330444] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.333721] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.336061] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.339934] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.347023] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.360096] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.367024] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.372223] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.380015] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.387018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.416017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.429151] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.440025] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.450024] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.475027] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.485623] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.499017] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.521015] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.574018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.597389] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.611015] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.620027] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.648017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.664383] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.676015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.682017] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.698019] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.711292] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.728017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.739020] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.777016] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.795724] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.798388] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.801418] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.804377] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.806219] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.810186] iommu: Default domain type: Passthrough [ 0.813451] SCSI subsystem initialized [ 0.815214] ACPI: bus type USB registered [ 0.817109] usbcore: registered new interface driver usbfs [ 0.818067] usbcore: registered new interface driver hub [ 0.820090] usbcore: registered new device driver usb [ 0.822198] pps_core: LinuxPPS API ver. 1 registered [ 0.825017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.828063] PTP clock support registered [ 0.830115] EDAC MC: Ver: 3.0.0 [ 0.831531] PCI: Using ACPI for IRQ routing [ 0.834026] NetLabel: Initializing [ 0.836010] NetLabel: domain hash size = 128 [ 0.837009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.840082] NetLabel: unlabeled traffic allowed by default [ 0.842471] vgaarb: loaded [ 0.844161] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.846013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.852402] clocksource: Switched to clocksource kvm-clock [ 0.964069] VFS: Disk quotas dquot_6.6.0 [ 0.966600] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.970360] *** VALIDATE ramfs *** [ 0.972142] *** VALIDATE hugetlbfs *** [ 0.974219] pnp: PnP ACPI init [ 0.977632] pnp: PnP ACPI: found 6 devices [ 0.993310] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.997181] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.999493] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.001908] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.004511] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.006992] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.010232] NET: Registered protocol family 2 [ 1.013040] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.017792] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.021424] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.027159] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.032107] TCP: Hash tables configured (established 65536 bind 65536) [ 1.036191] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.040443] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.044559] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.048840] NET: Registered protocol family 1 [ 1.052371] RPC: Registered named UNIX socket transport module. [ 1.055043] RPC: Registered udp transport module. [ 1.056765] RPC: Registered tcp transport module. [ 1.058581] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.061606] NET: Registered protocol family 44 [ 1.063832] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.066228] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.068660] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.071328] PCI: CLS 0 bytes, default 64 [ 1.073227] Unpacking initramfs... [ 2.482794] debug: unmapping init [mem 0xffff95edfcc54000-0xffff95edfffbffff] [ 2.486393] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.488581] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.491204] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.977265] Initialise system trusted keyrings [ 2.978697] Key type blacklist registered [ 2.981110] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.989686] zbud: loaded [ 2.993218] *** VALIDATE nfs *** [ 2.994770] *** VALIDATE nfs4 *** [ 2.996583] pstore: using deflate compression [ 3.001061] Platform Keyring initialized [ 3.115563] NET: Registered protocol family 38 [ 3.119971] Key type asymmetric registered [ 3.121226] Asymmetric key parser 'x509' registered [ 3.122735] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.125604] io scheduler mq-deadline registered [ 3.127269] io scheduler kyber registered [ 3.128626] io scheduler bfq registered [ 3.131058] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.133680] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.136761] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.139699] ACPI: Power Button [PWRF] [ 3.144937] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.151695] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.167487] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.174146] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.192297] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.221762] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.252154] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.257349] Non-volatile memory driver v1.3 [ 3.258934] Linux agpgart interface v0.103 [ 3.289867] virtio_blk virtio1: [vda] 149952 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.294208] vda: detected capacity change from 0 to 76775424 [ 3.310955] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.315245] vdb: detected capacity change from 0 to 1073741824 [ 3.335106] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.338410] vdc: detected capacity change from 0 to 2621440000 [ 3.354085] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.357357] vdd: detected capacity change from 0 to 2621440000 [ 3.376315] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.379436] vde: detected capacity change from 0 to 4294967296 [ 3.395933] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.399098] vdf: detected capacity change from 0 to 4294967296 [ 3.412657] libphy: Fixed MDIO Bus: probed [ 3.418328] usbcore: registered new interface driver usbserial_generic [ 3.420984] usbserial: USB Serial support registered for generic [ 3.423418] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.426954] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.428921] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.431773] mousedev: PS/2 mouse device common for all mice [ 3.434439] rtc_cmos 00:05: RTC can wake from S4 [ 3.437965] rtc_cmos 00:05: registered as rtc0 [ 3.440101] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.441744] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.445585] intel_pstate: CPU model not supported [ 3.452924] hid: raw HID events driver (C) Jiri Kosina [ 3.455465] usbcore: registered new interface driver usbhid [ 3.457960] usbhid: USB HID core driver [ 3.458606] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.460567] drop_monitor: Initializing network drop monitor service [ 3.466074] Initializing XFRM netlink socket [ 3.468697] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.468913] NET: Registered protocol family 10 [ 3.474421] Segment Routing with IPv6 [ 3.476054] NET: Registered protocol family 17 [ 3.478178] mpls_gso: MPLS GSO support [ 3.483991] RAS: Correctable Errors collector initialized. [ 3.486393] AVX version of gcm_enc/dec engaged. [ 3.488181] AES CTR mode by8 optimization enabled [ 3.564731] sched_clock: Marking stable (3564711800, 0)->(4489230090, -924518290) [ 3.568876] registered taskstats version 1 [ 3.571310] Loading compiled-in X.509 certificates [ 3.572876] zswap: loaded using pool lzo/zbud [ 3.598028] Key type big_key registered [ 3.610218] Key type encrypted registered [ 3.611632] ima: No TPM chip found, activating TPM-bypass! [ 3.614285] ima: Allocated hash algorithm: sha1 [ 3.616247] ima: No architecture policies found [ 3.618620] evm: Initialising EVM extended attributes: [ 3.620707] evm: security.selinux [ 3.621926] evm: security.ima [ 3.623064] evm: security.capability [ 3.626650] evm: HMAC attrs: 0x1 [ 3.629279] rtc_cmos 00:05: setting system clock to 2026-09-09 04:58:50 UTC (1788929930) [ 3.636503] debug: unmapping init [mem 0xffffffff9f403000-0xffffffff9f5fffff] [ 3.640160] debug: unmapping init [mem 0xffffffff9e182000-0xffffffff9e458fff] [ 3.647156] Write protecting the kernel read-only data: 28672k [ 3.650940] debug: unmapping init [mem 0xffffffff9c803000-0xffffffff9c9fffff] [ 3.654012] debug: unmapping init [mem 0xffffffff9d114000-0xffffffff9d1fffff] [ 3.688346] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.697596] systemd[1]: Detected virtualization kvm. [ 3.699448] systemd[1]: Detected architecture x86-64. [ 3.701459] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.728100] systemd[1]: No hostname configured. [ 3.729870] systemd[1]: Set hostname to . [ 3.731727] random: systemd: uninitialized urandom read (16 bytes read) [ 3.733811] systemd[1]: Initializing machine ID from random generator. [ 3.855242] random: systemd: uninitialized urandom read (16 bytes read) [ 3.857796] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 3.861639] random: systemd: uninitialized urandom read (16 bytes read) [ 3.864863] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.872140] systemd[1]: Reached target Paths. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Listening on udev Control Socket. [ OK ] Reached target Swap. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... Starting Setup Virtual Console... [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.508179] device-mapper: uevent: version 1.0.3 [ 4.511199] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ 5.362081] random: fast init done [ OK ] Reached target System Initialization. [ 5.376991] virtio_net virtio0 ens2: renamed from eth0 [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.410532] scsi host0: ata_piix [ 5.452421] scsi host1: ata_piix [ 5.454715] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.458031] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.891807] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.133196] random: crng init done [ 10.134688] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.725225] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.040567] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.298590] SELinux: Disabled at runtime. [ 12.362241] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.371882] systemd[1]: Detected virtualization kvm. [ 12.373948] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.870171] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.873956] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.879275] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.884623] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.888151] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.897563] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.925203] systemd[1]: Listening on Process Core Dump Socket. [ OK ] Listening on Process Core Dump Socket. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target rpc_pipefs.target. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Kernel Socket. Mounting Kernel Debug File System... [ OK ] Reached target RPC Port Mapper. [FAILED[ 13.002137] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS ] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-getty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.451272] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.763945] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.801385] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.976356] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.995180] EDAC sbridge: Ver: 1.1.2 [ 15.807895] Key type dns_resolver registered [ 16.135474] NFS: Registering the id_resolver key type [ 16.137179] Key type id_resolver registered [ 16.138638] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. Starting Network Manager... Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg413-server login: [ 43.431487] libcfs: loading out-of-tree module taints kernel. [ 43.458779] Key type ._llcrypt registered [ 43.459762] Key type .llcrypt registered [ 43.508649] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 52.749152] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 53.380808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.392075] alg: No test for adler32 (adler32-zlib) [ 54.424690] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 54.793219] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 56.440229] Key type lgssc registered [ 57.263410] Lustre: Echo OBD driver; http://www.lustre.org/ [ 72.014014] hrtimer: interrupt took 1983780 ns [ 75.502477] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 125.275707] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 142.437186] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 142.536320] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 143.805076] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 143.829321] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 143.888541] Lustre: lustre-MDT0000: new disk, initializing [ 143.973689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 143.992494] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 150.291098] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.539602] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 167.661344] Lustre: 6504:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 167.693563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 167.706766] Lustre: Skipped 1 previous similar message [ 167.878720] Lustre: lustre-MDT0001: new disk, initializing [ 168.027902] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 168.059508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 168.078256] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 173.440517] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 179.411845] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 191.760773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 191.991900] Lustre: lustre-OST0000: new disk, initializing [ 191.995638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 192.001421] Lustre: 8442:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 192.079412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 197.214153] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 197.243406] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 197.306302] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 199.934148] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 218.335929] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 218.587357] Lustre: lustre-OST0001: new disk, initializing [ 218.593943] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 218.601656] Lustre: 9519:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 218.692960] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.288409] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.445486] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.467597] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 228.555635] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 240.683412] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 250.151856] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 257.892141] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing check_logdir /tmp/testlogs/ [ 263.440373] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing yml_node [ 268.929884] Lustre: DEBUG MARKER: Client: 2.17.58.39 [ 272.106946] Lustre: DEBUG MARKER: MDS: 2.17.58.39 [ 275.078944] Lustre: DEBUG MARKER: OSS: 2.17.58.39 [ 277.493070] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 9 01:03:20 EDT 2026 [ 297.965143] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 299.452460] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 315.873362] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.873923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 315.883903] Lustre: Skipped 2 previous similar messages [ 315.885716] Lustre: Skipped 1 previous similar message [ 321.011455] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 321.019089] Lustre: Skipped 5 previous similar messages [ 321.553348] Lustre: server umount lustre-MDT0000 complete [ 325.156241] LustreError: 9518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788930252 with bad export cookie 17735062816091384885 [ 325.159238] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 325.166116] LustreError: 9518:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 326.114510] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.127699] Lustre: Skipped 1 previous similar message [ 326.136576] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.143046] Lustre: Skipped 1 previous similar message [ 330.149455] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 330.156963] Lustre: Skipped 1 previous similar message [ 331.550459] Lustre: server umount lustre-MDT0001 complete [ 336.871326] Lustre: server umount lustre-OST0000 complete [ 342.027752] Lustre: server umount lustre-OST0001 complete [ 350.479458] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 360.618535] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 405.999619] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 406.377104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 406.449591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 406.519675] Lustre: lustre-MDT0000: new disk, initializing [ 406.620084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 406.643768] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 411.210874] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 423.037147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 423.164823] Lustre: 15077:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 423.234388] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 423.238520] Lustre: Skipped 1 previous similar message [ 423.382435] Lustre: lustre-MDT0001: new disk, initializing [ 423.558468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 423.622144] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 423.648808] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 429.028751] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 440.183132] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 446.825745] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 454.738240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 455.202658] Lustre: lustre-OST0000: new disk, initializing [ 455.207642] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 455.220279] Lustre: 17056:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 455.344422] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 457.289275] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 457.297865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 457.503278] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 462.958364] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 474.691723] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 480.762193] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 481.101935] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 486.451549] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 486.726811] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 493.547920] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 493.554955] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 493.562377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 493.562770] LustreError: Skipped 1 previous similar message [ 493.577767] Lustre: Skipped 2 previous similar messages [ 495.578526] Lustre: server umount lustre-OST0000 complete [ 500.207512] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 500.215577] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 503.788296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 503.803405] Lustre: Skipped 2 previous similar messages [ 506.257928] Lustre: server umount lustre-MDT0000 complete [ 508.902058] LustreError: 15085:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 508.934885] LustreError: 15085:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 510.724520] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788930437 with bad export cookie 17735062816091386418 [ 510.725096] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 510.739796] LustreError: 15071:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 511.199388] Lustre: server umount lustre-MDT0001 complete [ 516.273281] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 01:07:20 (1788930440) [ 518.971195] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 520.894683] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 01:07:24 (1788930444) [ 529.046037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 529.632526] LustreError: 19465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 529.728532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 535.021570] LustreError: 19466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 535.095641] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 539.172532] LustreError: 19465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.225654] LustreError: 19466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.449310] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 549.891250] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 559.637534] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 565.796969] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 575.572387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 583.623559] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 593.384898] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 609.762193] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 609.771034] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 609.796607] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 609.801347] Lustre: Skipped 2 previous similar messages [ 609.835848] Lustre: Skipped 2 previous similar messages [ 614.389911] Lustre: server umount lustre-OST0000 complete [ 620.003859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 620.013215] Lustre: Skipped 1 previous similar message [ 633.312143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 633.663891] Lustre: server umount lustre-MDT0000 complete [ 634.352198] LustreError: 19466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 634.387092] LustreError: 19466:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 638.254530] LustreError: 21310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788930565 with bad export cookie 17735062816091387300 [ 638.275199] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 638.780148] Lustre: server umount lustre-MDT0001 complete [ 651.972902] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 655.500201] Key type lgssc unregistered [ 655.813213] LNet: 23022:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 655.821706] LNetError: 23022:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 655.845151] LNet: Removed LNI 192.168.204.113@tcp [ 656.727467] Key type .llcrypt unregistered [ 656.730346] Key type ._llcrypt unregistered [ 679.855254] Key type ._llcrypt registered [ 679.860090] Key type .llcrypt registered [ 679.963337] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 680.866450] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 680.895127] alg: No test for adler32 (adler32-zlib) [ 682.230790] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 682.484539] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 684.186190] Key type lgssc registered [ 685.320940] Lustre: Echo OBD driver; http://www.lustre.org/ [ 694.198405] loop: module loaded [ 694.306121] loop0: detected capacity change from 0 to 204800000 [ 694.614560] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 694.629722] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 695.052237] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 702.025884] loop0: detected capacity change from 0 to 204800000 [ 702.323886] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 702.334323] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 702.342796] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 708.555794] loop0: detected capacity change from 0 to 204800000 [ 714.636142] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 714.663981] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 716.094123] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 716.144292] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 716.257511] Lustre: lustre-MDT0000: new disk, initializing [ 716.360924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 716.391550] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 721.113650] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 730.456114] loop1: detected capacity change from 0 to 204800000 [ 737.057794] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 737.461573] Lustre: lustre-OST07c6: new disk, initializing [ 737.467947] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 737.475655] Lustre: Skipped 1 previous similar message [ 737.494917] Lustre: 26211:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 737.594820] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 744.984953] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 746.619662] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 746.652338] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 746.695716] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 750.337903] Lustre: 25198:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 750.345813] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 750.355808] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 750.369712] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 750.377359] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 750.383944] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 763.606088] Lustre: 25197:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 763.617638] Lustre: 25197:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 763.624333] Lustre: 25197:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 763.633784] Lustre: 25197:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 763.640263] Lustre: 25197:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 763.647691] Lustre: 25197:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 769.276030] Lustre: 25196:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 769.285449] Lustre: 25196:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 769.295289] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 769.309851] Lustre: 25196:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.318682] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 769.331343] Lustre: 25196:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.341764] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 769.354566] Lustre: 25196:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.362462] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 769.370494] Lustre: 25196:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 769.378798] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 769.386520] Lustre: 25196:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.640884] Lustre: 25198:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 772.654121] Lustre: 25198:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 772.663258] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 772.669619] Lustre: 25198:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.683301] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 772.690522] Lustre: 25198:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.697872] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 772.704299] Lustre: 25198:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.712774] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 772.718607] Lustre: 25198:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 772.724529] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 772.730293] Lustre: 25198:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 781.222753] Lustre: 26219:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 781.233575] Lustre: 26219:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 781.239177] Lustre: 26219:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 781.255399] Lustre: 26219:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 781.263584] Lustre: 26219:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 781.278767] Lustre: 26219:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 781.295330] Lustre: 26219:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 781.302599] Lustre: 26219:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 781.307566] Lustre: 26219:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 781.317855] Lustre: 26219:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 781.328833] Lustre: 26219:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 781.338149] Lustre: 26219:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 792.791645] Lustre: 26219:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 792.818501] Lustre: 26219:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 792.832711] Lustre: 26219:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 792.848324] Lustre: 26219:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.861579] Lustre: 26219:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 792.871461] Lustre: 26219:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.889992] Lustre: 26219:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 792.913459] Lustre: 26219:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.924322] Lustre: 26219:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 792.940921] Lustre: 26219:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 792.957368] Lustre: 26219:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 792.964587] Lustre: 26219:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 843.745400] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 843.757082] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 843.783278] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 845.931910] Lustre: server umount lustre-OST07c6 complete [ 857.719102] Lustre: server umount lustre-MDT0000 complete [ 877.708980] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 880.772664] Key type lgssc unregistered [ 881.032057] LNet: 28765:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 881.041224] LNetError: 28765:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 881.066250] LNet: Removed LNI 192.168.204.113@tcp [ 882.131170] Key type .llcrypt unregistered [ 882.132754] Key type ._llcrypt unregistered [ 903.898145] Key type ._llcrypt registered [ 903.899554] Key type .llcrypt registered [ 904.038619] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 904.881915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 904.944745] alg: No test for adler32 (adler32-zlib) [ 906.236302] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 906.565619] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 908.316636] Key type lgssc registered [ 909.728890] Lustre: Echo OBD driver; http://www.lustre.org/ [ 917.613223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 917.629372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 919.007601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 923.631559] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 928.228082] LustreError: 30035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 928.247716] LustreError: 30035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 932.205508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 932.649053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 938.507991] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 947.133618] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 953.308675] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 961.135992] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 961.597314] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 963.640802] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 968.401566] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 976.811867] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 994.278116] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 994.288325] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 994.315747] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 995.812822] Lustre: server umount lustre-OST0000 complete [ 1003.509662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1003.515655] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1003.522988] Lustre: Skipped 1 previous similar message [ 1003.540335] Lustre: Skipped 1 previous similar message [ 1006.165094] Lustre: server umount lustre-MDT0000 complete [ 1008.610497] LustreError: 30034:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1008.645359] LustreError: 30034:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1010.020768] LustreError: 30016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788930936 with bad export cookie 10560486147547347644 [ 1010.029165] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1010.035962] LustreError: 30016:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1019.663723] Lustre: server umount lustre-MDT0001 complete [ 1031.174672] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1034.395366] Key type lgssc unregistered [ 1034.760565] LNet: 33771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1034.768469] LNetError: 33771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1034.785859] LNet: Removed LNI 192.168.204.113@tcp [ 1035.670804] Key type .llcrypt unregistered [ 1035.672847] Key type ._llcrypt unregistered [ 1057.986319] Key type ._llcrypt registered [ 1057.990370] Key type .llcrypt registered [ 1058.164830] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1059.110821] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1059.169388] alg: No test for adler32 (adler32-zlib) [ 1060.218235] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 1060.532188] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1062.249302] Key type lgssc registered [ 1063.382542] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1071.686904] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1071.709250] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1073.038506] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1077.669525] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1082.363281] LustreError: 35038:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1082.386513] LustreError: 35038:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1087.467780] LustreError: 35039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1087.666432] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1088.049157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1093.028294] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1102.337591] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1112.413423] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1121.840234] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1122.068424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1129.329421] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1138.294690] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1142.601265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1158.128750] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1158.139991] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1158.152153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1160.497188] Lustre: server umount lustre-OST0000 complete [ 1168.357785] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.369442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1168.386835] Lustre: Skipped 2 previous similar messages [ 1168.398884] Lustre: Skipped 2 previous similar messages [ 1171.654261] Lustre: server umount lustre-MDT0000 complete [ 1173.474718] LustreError: 35039:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1173.496978] LustreError: 35039:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1176.720279] LustreError: 35018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788931103 with bad export cookie 17505882366947649678 [ 1176.720898] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1176.751902] LustreError: 35018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1177.442473] Lustre: server umount lustre-MDT0001 complete [ 1191.256236] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1194.631178] Key type lgssc unregistered [ 1195.002755] LNet: 38596:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1195.016431] LNetError: 38596:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1196.077938] LNet: Removed LNI 192.168.204.113@tcp [ 1197.364757] Key type .llcrypt unregistered [ 1197.368748] Key type ._llcrypt unregistered [ 1225.004309] Key type ._llcrypt registered [ 1225.007557] Key type .llcrypt registered [ 1225.155813] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1227.023340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1227.074325] alg: No test for adler32 (adler32-zlib) [ 1228.254936] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 1228.525663] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1230.185371] Key type lgssc registered [ 1231.671525] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1240.103869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1240.131092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1241.765063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.099307] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1251.301872] LustreError: 40011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1251.330856] LustreError: 40011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1255.102342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1255.698322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1262.467388] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1271.417669] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1278.721713] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1286.525772] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1286.637746] LustreError: 41830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.935446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1292.271249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1294.512483] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1304.870783] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1316.836607] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1316.848080] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1316.856453] Lustre: Skipped 2 previous similar messages [ 1316.877119] Lustre: Skipped 2 previous similar messages [ 1317.376429] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 1318.715654] Lustre: server umount lustre-MDT0000 complete [ 1321.955671] LustreError: 40006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1321.998481] LustreError: 40006:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1323.347780] LustreError: 39993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788931250 with bad export cookie 13939027216294963861 [ 1323.353315] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1323.359233] LustreError: 39993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1327.074889] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1327.085569] LustreError: 40006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1327.087096] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1327.114081] LustreError: 40006:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1329.987238] Lustre: server umount lustre-MDT0001 complete [ 1346.620784] Lustre: server umount lustre-OST0000 complete [ 1369.044553] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1372.672332] Key type lgssc unregistered [ 1373.009178] LNet: 43877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1373.021924] LNetError: 43877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1373.048282] LNet: Removed LNI 192.168.204.113@tcp [ 1374.247237] Key type .llcrypt unregistered [ 1374.249042] Key type ._llcrypt unregistered [ 1407.482861] Key type ._llcrypt registered [ 1407.484557] Key type .llcrypt registered [ 1407.670217] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1409.305451] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1409.409864] alg: No test for adler32 (adler32-zlib) [ 1410.672441] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 1411.051980] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1412.832252] Key type lgssc registered [ 1414.622547] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1424.471479] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1424.491277] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1425.912350] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1431.466825] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1436.140474] LustreError: 45291:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.167422] LustreError: 45291:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1441.262661] LustreError: 45292:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.842799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1442.213585] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1446.918174] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1455.634682] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1463.038531] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1470.838630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1471.293319] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1472.328926] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1478.564229] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1487.283809] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1494.457296] Lustre: lustre-OST0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 1498.095861] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1498.099993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1498.132260] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1499.577281] Lustre: lustre-OST0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 1499.589622] Lustre: Skipped 1 previous similar message [ 1503.214664] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1508.328467] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1508.332733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1508.343575] Lustre: Skipped 2 previous similar messages [ 1508.521632] Lustre: server umount lustre-OST0000 complete [ 1528.804624] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1528.813599] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1528.816220] Lustre: Skipped 2 previous similar messages [ 1528.833350] Lustre: Skipped 2 previous similar messages [ 1532.752358] Lustre: server umount lustre-MDT0000 complete [ 1533.924547] LustreError: 46481:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1533.941473] LustreError: 46481:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1536.695652] LustreError: 45272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788931463 with bad export cookie 5199141850245191926 [ 1536.701942] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1536.712107] LustreError: 45272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1543.175987] Lustre: server umount lustre-MDT0001 complete [ 1554.313879] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1557.022526] Key type lgssc unregistered [ 1557.325780] LNet: 49000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1557.336589] LNetError: 49000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1558.377390] LNet: Removed LNI 192.168.204.113@tcp [ 1559.173198] Key type .llcrypt unregistered [ 1559.177440] Key type ._llcrypt unregistered [ 1581.613746] Key type ._llcrypt registered [ 1581.615479] Key type .llcrypt registered [ 1581.770183] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1582.717654] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1582.732193] alg: No test for adler32 (adler32-zlib) [ 1583.855298] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 1584.104433] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1585.760261] Key type lgssc registered [ 1586.887785] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1594.688801] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1594.702841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1596.128331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1600.420441] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1605.605192] LustreError: 50412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1605.620833] LustreError: 50412:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1610.275412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1610.735165] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1615.487632] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1625.992834] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1632.432931] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1639.694908] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1640.008094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1641.088051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1645.965850] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1654.047870] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1660.216273] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1664.036817] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1676.559924] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-09 1h26m40s [ 1678.434747] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1681.379441] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1681.389887] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1681.413427] Lustre: Skipped 1 previous similar message [ 1682.476592] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1682.485077] Lustre: Skipped 4 previous similar messages [ 1683.957745] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 1683.968751] Lustre: Skipped 1 previous similar message [ 1686.834806] Lustre: server umount lustre-MDT0000 complete [ 1688.563271] LustreError: 50406:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1688.574032] LustreError: 50406:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1690.422900] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1694.693375] LustreError: 50412:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1694.715816] LustreError: 50412:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 1697.451391] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1697.617157] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1697.934294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1697.991113] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1698.998790] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1703.327643] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1707.251907] Lustre: DEBUG MARKER: Wait for df (36787) ... [ 1709.159839] Lustre: DEBUG MARKER: done [ 1714.666667] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1714.675473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1714.699997] Lustre: Skipped 1 previous similar message [ 1714.710332] Lustre: Skipped 1 previous similar message [ 1719.665958] Lustre: server umount lustre-OST0000 complete [ 1724.901427] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1724.905420] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1724.915336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1724.918972] Lustre: Skipped 1 previous similar message [ 1730.747788] Lustre: server umount lustre-MDT0000 complete [ 1732.078993] LustreError: 51399:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1732.115940] LustreError: 51399:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1735.306240] LustreError: 50391:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788931662 with bad export cookie 14176909977186541853 [ 1735.314093] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1735.721312] Lustre: server umount lustre-MDT0001 complete [ 1748.178163] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1751.650443] Key type lgssc unregistered [ 1752.096101] LNet: 55335:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1752.107464] LNetError: 55335:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1752.138160] LNet: Removed LNI 192.168.204.113@tcp [ 1753.027051] Key type .llcrypt unregistered [ 1753.030398] Key type ._llcrypt unregistered [ 1797.151339] Key type ._llcrypt registered [ 1797.154339] Key type .llcrypt registered [ 1797.255463] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 1798.230439] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1798.266149] alg: No test for adler32 (adler32-zlib) [ 1799.284438] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 1799.528290] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 1801.208296] Key type lgssc registered [ 1802.121652] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1809.303847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1809.321441] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1810.576161] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1810.594409] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1810.716096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1815.057952] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1822.883465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1823.032059] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1823.108175] Lustre: 57539:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1823.419371] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1827.976373] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1834.797755] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1840.203798] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1846.321573] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1846.478309] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1846.651482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1852.206390] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 1855.147924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1859.992568] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1866.735232] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-09 1h29m51s [ 1868.115217] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1875.089442] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1877.355603] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1877.405771] LustreError: 57545:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1897.951940] Lustre: lustre-MDT0000: Client 254830c8-c94a-45d4-89a0-29f06d2b2ffd (at 192.168.204.13@tcp) reconnecting [ 1917.488145] LustreError: 57545:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1919.642815] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1922.386247] Lustre: DEBUG MARKER: done [ 1930.186734] Lustre: server umount lustre-OST0000 complete [ 1936.355369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1936.361352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1936.368582] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1937.442444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1937.446749] Lustre: Skipped 2 previous similar messages [ 1943.534581] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1943.545639] Lustre: Skipped 1 previous similar message [ 1943.601448] Lustre: server umount lustre-MDT0000 complete [ 1946.720633] LustreError: 59400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788931873 with bad export cookie 15522721027021846057 [ 1946.723381] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1946.730220] LustreError: 59400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1956.515462] Lustre: server umount lustre-MDT0001 complete [ 1965.869941] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 1968.228856] Key type lgssc unregistered [ 1968.513574] LNet: 61869:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1968.524228] LNetError: 61869:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1968.539598] LNet: Removed LNI 192.168.204.113@tcp [ 1969.228227] Key type .llcrypt unregistered [ 1969.230465] Key type ._llcrypt unregistered [ 2011.544952] Key type ._llcrypt registered [ 2011.549517] Key type .llcrypt registered [ 2011.676180] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2012.815832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2012.847173] alg: No test for adler32 (adler32-zlib) [ 2014.004928] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 2014.300732] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2016.013953] Key type lgssc registered [ 2017.346898] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2020.711980] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2020.740750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2022.013551] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2022.025878] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2022.210866] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2023.146574] Lustre: Failing over lustre-MDT0000 [ 2023.437764] Lustre: server umount lustre-MDT0000 complete [ 2030.733053] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:32:34 (1788931954) [ 2037.287242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2037.713973] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2042.678205] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2051.253211] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2051.393748] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2051.440516] Lustre: 64667:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2051.783016] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2055.337563] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2063.360958] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2069.301669] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2077.659694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2077.820132] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2078.035121] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2084.938891] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2086.580918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2092.180690] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2097.356512] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2107.367469] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2107.371215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2107.385772] Lustre: Skipped 2 previous similar messages [ 2107.404700] Lustre: Skipped 2 previous similar messages [ 2109.310074] Lustre: server umount lustre-MDT0000 complete [ 2112.607490] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788932039 with bad export cookie 12075829604993497946 [ 2112.616221] LustreError: 64657:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2112.616986] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2112.822779] Lustre: server umount lustre-MDT0001 complete [ 2116.264904] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2120.448699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2129.089372] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.984544] Lustre: 63590:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788932044/real 1788932044] req@00000000a71ab472 x1875831099008256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788932060 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2134.023803] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2138.145319] LustreError: 63587:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f59da2fe x1875831099009024/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2138.853202] LustreError: 66514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.883234] LustreError: 66514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2138.958562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2139.066073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2139.955168] Lustre: 63589:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788932049/real 1788932049] req@00000000d11d26d4 x1875831099008512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788932065 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2140.062180] LustreError: 66515:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2143.085965] LustreError: 66514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2144.819186] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2145.127051] Lustre: 63588:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788932054/real 1788932054] req@00000000cbab6e51 x1875831099008640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788932070 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2148.195670] Lustre: 63590:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788932059/real 1788932059] req@000000003ca544f2 x1875831099008768/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788932075 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2149.353318] LustreError: 68653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2149.390112] LustreError: 68653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2154.470500] LustreError: 68653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2154.490929] LustreError: 68653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2157.619923] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2158.277596] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2163.374475] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2163.816328] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2173.216320] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2179.312801] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2188.777486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2188.797317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2188.829251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.091823] Lustre: server umount lustre-MDT0000 complete [ 2194.406838] LustreError: 68653:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.431500] LustreError: 68653:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2199.103821] LustreError: 66520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788932125 with bad export cookie 12075829604993500221 [ 2199.105273] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2199.129616] LustreError: 66520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2199.523037] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2199.537025] Lustre: Skipped 2 previous similar messages [ 2200.044475] Lustre: server umount lustre-MDT0001 complete [ 2208.292511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2218.238501] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2225.121161] LustreError: 63587:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000060e88b9b x1875831099037056/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2225.538310] LustreError: 66514:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.561359] LustreError: 66514:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2225.661344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2225.753596] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2229.739495] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2229.742675] Lustre: Skipped 1 previous similar message [ 2230.221431] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2239.659630] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2241.004582] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2244.592945] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2254.549623] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2262.159692] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2271.722525] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2271.725619] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2271.746384] Lustre: Skipped 2 previous similar messages [ 2271.770855] Lustre: Skipped 5 previous similar messages [ 2276.837210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2276.843204] Lustre: Skipped 2 previous similar messages [ 2278.026457] Lustre: server umount lustre-MDT0000 complete [ 2281.954677] LustreError: 71851:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2281.986043] LustreError: 71851:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2282.187427] LustreError: 66520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788932209 with bad export cookie 12075829604993501600 [ 2282.191643] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2282.202072] LustreError: 66520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2282.430599] Lustre: server umount lustre-MDT0001 complete [ 2289.396487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2293.826842] Lustre: DEBUG MARKER: files compared the same [ 2306.268362] Lustre: server umount lustre-OST0000 complete [ 2325.761438] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2329.921804] Key type lgssc unregistered [ 2330.291756] LNet: 74394:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2330.305366] LNetError: 74394:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2330.344647] LNet: Removed LNI 192.168.204.113@tcp [ 2331.594798] Key type .llcrypt unregistered [ 2331.597076] Key type ._llcrypt unregistered [ 2362.148564] Key type ._llcrypt registered [ 2362.152131] Key type .llcrypt registered [ 2362.240635] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2363.047103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2363.193867] alg: No test for adler32 (adler32-zlib) [ 2364.216870] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 2364.484396] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2366.136275] Key type lgssc registered [ 2367.490295] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2376.638750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2376.656695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2378.195916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2384.842894] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug +malloc all [ 2388.452030] LustreError: 75810:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2388.463592] LustreError: 75810:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2393.587719] LustreError: 75811:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2395.407099] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2395.776767] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2400.758718] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug +malloc all [ 2408.988410] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2415.467682] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2424.074821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2424.467673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2431.003567] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug +malloc all [ 2440.424344] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2449.064861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2459.625318] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2459.632990] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2459.633379] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2464.739825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2464.751446] Lustre: Skipped 1 previous similar message [ 2465.579687] Lustre: server umount lustre-OST0000 complete [ 2469.857515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2469.862443] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2469.881649] Lustre: Skipped 2 previous similar messages [ 2469.904717] Lustre: Skipped 1 previous similar message [ 2474.983190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2474.992592] Lustre: Skipped 1 previous similar message [ 2475.894248] Lustre: server umount lustre-MDT0000 complete [ 2479.627614] LustreError: 75790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788932406 with bad export cookie 7225964646059948606 [ 2479.634358] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2479.639256] LustreError: 75790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2480.136927] Lustre: server umount lustre-MDT0001 complete [ 2492.963164] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2496.013710] Key type lgssc unregistered [ 2496.282683] LNet: 79367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2496.288711] LNetError: 79367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2496.307238] LNet: Removed LNI 192.168.204.113@tcp [ 2497.325879] Key type .llcrypt unregistered [ 2497.328262] Key type ._llcrypt unregistered [ 2524.436962] Key type ._llcrypt registered [ 2524.439747] Key type .llcrypt registered [ 2524.522857] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2525.473226] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2525.484813] alg: No test for adler32 (adler32-zlib) [ 2526.579614] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 2526.889944] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2528.536628] Key type lgssc registered [ 2530.035735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2539.609658] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2539.635892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2555.296440] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2560.416445] LustreError: 80755:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [ 2587.104204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2594.897594] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2602.240819] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2610.803850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2610.968216] Lustre: *** cfs_fail_loc=706, val=0*** [ 2611.419809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2611.518992] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2616.802124] LustreError: 80780:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2616.829531] LustreError: 80780:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2617.174745] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2621.926993] LustreError: 80781:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2621.951757] LustreError: 80781:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2627.050266] LustreError: 81840:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2627.069218] LustreError: 81840:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2627.335033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2627.849607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2632.429953] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2641.416353] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2647.887527] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2652.642636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2652.655912] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2652.700076] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2653.665952] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2653.670528] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2653.682570] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2656.834120] Lustre: server umount lustre-OST0000 complete [ 2663.915446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2663.916226] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.927050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2663.927059] Lustre: Skipped 1 previous similar message [ 2663.961877] Lustre: Skipped 1 previous similar message [ 2667.806590] Lustre: server umount lustre-MDT0000 complete [ 2669.027280] LustreError: 81845:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2669.048452] LustreError: 81845:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2671.910548] LustreError: 80761:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788932598 with bad export cookie 7003369623987764588 [ 2671.929059] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2672.627308] Lustre: server umount lustre-MDT0001 complete [ 2685.614335] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 2688.558442] Key type lgssc unregistered [ 2689.002903] LNet: 84368:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2689.013373] LNetError: 84368:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2689.038423] LNet: Removed LNI 192.168.204.113@tcp [ 2690.091326] Key type .llcrypt unregistered [ 2690.096191] Key type ._llcrypt unregistered [ 2719.180927] Key type ._llcrypt registered [ 2719.182851] Key type .llcrypt registered [ 2719.344639] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2720.769289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2720.892993] alg: No test for adler32 (adler32-zlib) [ 2721.913659] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 2722.167850] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 2723.844495] Key type lgssc registered [ 2725.501868] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2736.022825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2736.043478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2743.287365] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2752.730711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2753.553649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2758.630727] LustreError: 86468:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.645344] LustreError: 86468:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2758.753857] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2763.748305] LustreError: 86467:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.775887] LustreError: 86467:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2768.867764] LustreError: 86471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.894537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2768.912699] LustreError: 86471:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2769.467345] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2774.503302] LustreError: 86472:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2774.532860] LustreError: 86472:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 2775.871572] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2778.597132] LustreError: 86471:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2778.611173] LustreError: 86471:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2784.824843] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2784.838704] Lustre: MGS: Client c437c150-9071-4c42-a12a-615cfa6d2cec (at 0@lo) reconnecting [ 2784.875942] LustreError: 86482:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2785.148755] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2785.216988] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2789.731376] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2805.734895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2805.747257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2805.783405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2810.072596] Lustre: server umount lustre-OST0000 complete [ 2815.970451] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2815.977447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2815.984080] Lustre: Skipped 2 previous similar messages [ 2815.995353] Lustre: Skipped 1 previous similar message [ 2820.916987] Lustre: server umount lustre-MDT0000 complete [ 2821.100868] LustreError: 88014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.122128] LustreError: 88014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2825.624951] Lustre: server umount lustre-MDT0001 complete [ 2830.147418] Lustre: server umount MGS complete [ 2844.435869] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:46:08 (1788932768) [ 2877.794221] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 2891.163545] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 2928.628456] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2934.036921] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2944.831741] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2945.194222] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2945.238279] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2945.332222] Lustre: lustre-MDT0001: new disk, initializing [ 2945.426827] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2945.469448] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2945.478371] Lustre: 93643:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2945.492914] LustreError: 93643:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2950.263441] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2961.824454] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2962.085311] Lustre: lustre-OST0000: new disk, initializing [ 2962.088907] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2962.096413] Lustre: 94408:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2962.206545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2968.695832] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2971.748341] Lustre: 94864:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000d74f2e58 x1875831841399424/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788932914 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2978.660826] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2986.161837] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2986.178342] Lustre: MGS: Client 6efc5f4c-1348-4eef-9e0d-c326a01db2bd (at 0@lo) reconnecting [ 2986.195385] LustreError: 93629:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2986.231627] Lustre: 92909:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2986.297323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2986.406715] Lustre: lustre-MDT0000: new disk, initializing [ 2986.549761] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2986.579677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2986.615870] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2987.686405] Lustre: 95477:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000049f4b4f0 x1875831841406336/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788932930 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2991.553814] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 2992.874602] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2992.882574] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2992.911238] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 2996.792222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 2996.808115] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3023.847668] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3023.861648] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3023.868094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3023.879428] Lustre: Skipped 1 previous similar message [ 3023.883860] Lustre: Skipped 1 previous similar message [ 3028.401581] Lustre: server umount lustre-OST0000 complete [ 3034.086899] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3034.087562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3034.131448] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3034.142211] Lustre: Skipped 2 previous similar messages [ 3038.912485] Lustre: server umount lustre-MDT0000 complete [ 3039.201773] LustreError: 93613:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3039.232299] LustreError: 93613:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3049.544257] Lustre: server umount lustre-MDT0001 complete [ 3054.376762] Lustre: server umount MGS complete [ 3072.592126] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:49:55 (1788932995) [ 3094.738907] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3098.256120] Key type lgssc unregistered [ 3098.669083] LNet: 97975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3098.682886] LNetError: 97975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3098.721710] LNet: Removed LNI 192.168.204.113@tcp [ 3099.798183] Key type .llcrypt unregistered [ 3099.802839] Key type ._llcrypt unregistered [ 3121.531523] Key type ._llcrypt registered [ 3121.533682] Key type .llcrypt registered [ 3121.666919] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3122.916311] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3122.945163] alg: No test for adler32 (adler32-zlib) [ 3124.017818] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 3124.379338] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3126.104567] Key type lgssc registered [ 3127.191700] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3135.278128] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3135.320303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3137.186991] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3142.977982] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3146.726392] LustreError: 99197:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.766939] LustreError: 99197:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3151.869981] LustreError: 99196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3153.196115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3153.768221] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3158.796784] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3168.505404] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3176.819744] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3184.899337] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3185.424933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3191.815945] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3200.721378] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3206.908593] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3220.453369] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3220.463193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3220.482954] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3222.497655] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3222.508142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3222.517961] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3222.551446] Lustre: Skipped 1 previous similar message [ 3225.264455] Lustre: server umount lustre-OST0000 complete [ 3230.705850] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3230.723826] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3230.758923] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3232.739422] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3232.763183] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3232.768463] Lustre: Skipped 1 previous similar message [ 3236.252677] Lustre: server umount lustre-MDT0000 complete [ 3237.864471] LustreError: 101690:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.886727] LustreError: 101690:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3241.033913] LustreError: 99177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933167 with bad export cookie 9172619176592027837 [ 3241.045246] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3241.052892] LustreError: 99177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3241.464777] Lustre: server umount lustre-MDT0001 complete [ 3256.510071] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3259.878954] Key type lgssc unregistered [ 3260.193734] LNet: 102753:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3260.211096] LNetError: 102753:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3260.241176] LNet: Removed LNI 192.168.204.113@tcp [ 3261.044591] Key type .llcrypt unregistered [ 3261.049887] Key type ._llcrypt unregistered [ 3278.190777] Key type ._llcrypt registered [ 3278.195733] Key type .llcrypt registered [ 3278.312400] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3279.607510] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3279.648433] alg: No test for adler32 (adler32-zlib) [ 3280.731745] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 3281.116622] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3282.873249] Key type lgssc registered [ 3284.058680] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3291.448189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3291.454757] LustreError: 103832:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3291.474587] LustreError: 103833:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3291.483961] LustreError: 103832:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4977 [ 3291.495905] LustreError: 103832:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3291.501184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3291.504293] LustreError: 103832:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3292.972821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3301.507912] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3301.858612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3307.507421] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3314.225187] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing lsmod [ 3317.112931] LustreError: 105029:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3317.207184] LustreError: 105040:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3317.227025] LustreError: 105029:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4896 [ 3317.236374] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3317.243513] LustreError: 105029:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3317.243552] LustreError: 105029:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3317.815980] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3322.342945] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3322.352193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3322.368210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3322.853719] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3322.865302] Lustre: Skipped 1 previous similar message [ 3322.869830] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3322.875487] Lustre: Skipped 1 previous similar message [ 3327.014673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3327.028558] Lustre: Skipped 2 previous similar messages [ 3327.355276] Lustre: server umount lustre-MDT0000 complete [ 3331.674907] LustreError: 105079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933258 with bad export cookie 12011613345239002194 [ 3331.679367] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3331.688791] LustreError: 105079:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3331.703319] Lustre: 103863:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.204.113@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 3332.065670] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3332.069623] LustreError: 103867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3332.083225] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3332.119941] LustreError: 103867:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3333.216302] LustreError: 104301:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000f949339 x1875832426483456/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3333.410930] Lustre: server umount lustre-MDT0001 complete [ 3337.385358] Lustre: server umount lustre-OST0000 complete [ 3346.962103] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3347.572260] LustreError: 106103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3347.644063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3352.676928] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3353.064498] LustreError: 106104:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3357.165923] LustreError: 106103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.278738] LustreError: 106104:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3362.331633] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3362.772266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3368.844822] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3377.978974] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3385.504801] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3392.981608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3393.442208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3399.529958] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3407.096660] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3418.055253] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 3428.328781] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3428.329779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3428.331307] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3428.337930] LustreError: Skipped 1 previous similar message [ 3428.357765] Lustre: Skipped 1 previous similar message [ 3431.448737] Lustre: server umount lustre-OST0000 complete [ 3438.561466] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3438.563860] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3438.595833] Lustre: Skipped 2 previous similar messages [ 3438.620918] Lustre: Skipped 1 previous similar message [ 3441.902856] Lustre: server umount lustre-MDT0000 complete [ 3443.683843] LustreError: 106103:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3443.703181] LustreError: 106103:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3446.209637] LustreError: 106838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933373 with bad export cookie 12011613345239002887 [ 3446.212392] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3446.230100] LustreError: 106838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3446.680079] Lustre: server umount lustre-MDT0001 complete [ 3459.133527] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3462.059837] Key type lgssc unregistered [ 3462.381470] LNet: 109657:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3462.397585] LNetError: 109657:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3462.423854] LNet: Removed LNI 192.168.204.113@tcp [ 3463.278728] Key type .llcrypt unregistered [ 3463.283077] Key type ._llcrypt unregistered [ 3490.874600] Key type ._llcrypt registered [ 3490.879400] Key type .llcrypt registered [ 3491.056505] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3492.670933] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3492.722596] alg: No test for adler32 (adler32-zlib) [ 3493.811972] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 3494.009818] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3495.761367] Key type lgssc registered [ 3497.213274] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3506.040207] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3506.056514] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3507.703236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3513.341698] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3517.927048] LustreError: 111072:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3517.970840] LustreError: 111072:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3523.044582] LustreError: 111071:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3525.728792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3526.582196] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3531.223713] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3540.365783] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3548.361166] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3556.014567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3556.511646] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3564.867881] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3574.416415] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3578.026664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 3585.018872] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3590.945370] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3597.797160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3597.808797] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.841454] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3598.820555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3598.835834] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3598.851545] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3602.908158] Lustre: server umount lustre-OST0000 complete [ 3608.553971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3608.567666] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3608.579788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3608.590719] Lustre: Skipped 1 previous similar message [ 3613.578830] Lustre: server umount lustre-MDT0000 complete [ 3614.184974] LustreError: 112258:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3614.213309] LustreError: 112258:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3618.211543] LustreError: 111052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933545 with bad export cookie 7048526290269740177 [ 3618.219773] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3618.223376] LustreError: 111052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3618.892639] Lustre: server umount lustre-MDT0001 complete [ 3631.673465] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3634.813535] Key type lgssc unregistered [ 3635.237549] LNet: 114713:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3635.250504] LNetError: 114713:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3635.269137] LNet: Removed LNI 192.168.204.113@tcp [ 3636.128135] Key type .llcrypt unregistered [ 3636.129664] Key type ._llcrypt unregistered [ 3660.434150] Key type ._llcrypt registered [ 3660.440438] Key type .llcrypt registered [ 3660.736561] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3662.573608] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3662.627646] alg: No test for adler32 (adler32-zlib) [ 3663.693877] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 3664.054942] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3665.730361] Key type lgssc registered [ 3667.108567] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3676.793331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3676.832702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3678.340617] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3684.026316] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3688.942684] LustreError: 115933:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3688.959684] LustreError: 115933:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3694.064336] LustreError: 115932:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3694.227228] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3694.974843] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3700.008781] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3711.522272] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3717.989534] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3727.170146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3728.004883] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3736.764927] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3746.019992] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3748.501981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 3764.192972] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3764.199734] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3764.212958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3764.227869] Lustre: Skipped 1 previous similar message [ 3764.258347] Lustre: Skipped 1 previous similar message [ 3766.644550] Lustre: server umount lustre-OST0000 complete [ 3771.873304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3771.889256] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3771.908660] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3774.434903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3774.435707] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3777.673859] Lustre: server umount lustre-MDT0000 complete [ 3779.553635] LustreError: 115927:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3779.577360] LustreError: 115927:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3782.714264] LustreError: 115913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933709 with bad export cookie 2317955693472984221 [ 3782.725630] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3782.731270] LustreError: 115913:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3783.622528] Lustre: server umount lustre-MDT0001 complete [ 3797.463887] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 3800.392523] Key type lgssc unregistered [ 3800.706388] LNet: 119508:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3800.716590] LNetError: 119508:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3800.737799] LNet: Removed LNI 192.168.204.113@tcp [ 3801.649352] Key type .llcrypt unregistered [ 3801.653443] Key type ._llcrypt unregistered [ 3826.508844] Key type ._llcrypt registered [ 3826.510807] Key type .llcrypt registered [ 3826.620801] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 3827.349997] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3827.452621] alg: No test for adler32 (adler32-zlib) [ 3828.537353] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 3828.780901] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 3830.456384] Key type lgssc registered [ 3831.677621] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3839.559164] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3839.581976] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3841.173560] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3846.371796] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3851.749035] LustreError: 120920:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3851.769985] LustreError: 120920:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3856.352846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3857.026513] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3862.243609] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3871.038408] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3878.054750] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3886.409356] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3886.702492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3887.766868] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 3896.617648] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 3905.719228] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3914.036676] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3918.096907] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3918.103799] Lustre: Skipped 2 previous similar messages [ 3921.455622] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3921.473547] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3922.321582] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3922.329472] Lustre: Skipped 2 previous similar messages [ 3929.173356] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3929.185592] Lustre: Skipped 1 previous similar message [ 3935.663378] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.204.13@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3935.670731] Lustre: Skipped 2 previous similar messages [ 3937.438245] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.204.13@tcp 192.168.0.[2,10]@tcp [ 3949.130047] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.204.13/24@tcp in log lustre-MDT0000 [ 3949.137755] Lustre: Skipped 2 previous similar messages [ 3957.293610] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.13/24@tcp [ 3957.297691] Lustre: Skipped 1 previous similar message [ 3969.001975] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3969.008226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3969.049734] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3969.058764] Lustre: Skipped 1 previous similar message [ 3971.316877] Lustre: server umount lustre-OST0000 complete [ 3979.233389] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3979.253866] Lustre: Skipped 1 previous similar message [ 3979.262963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3979.277251] Lustre: Skipped 1 previous similar message [ 3981.872118] Lustre: server umount lustre-MDT0000 complete [ 3984.354310] LustreError: 121650:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3986.230027] LustreError: 123553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788933913 with bad export cookie 18207025427226478085 [ 3986.233357] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3986.247162] LustreError: 123553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3986.777781] Lustre: server umount lustre-MDT0001 complete [ 3999.442204] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4002.668786] Key type lgssc unregistered [ 4003.247951] LNet: 125915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4003.262596] LNetError: 125915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4003.296501] LNet: Removed LNI 192.168.204.113@tcp [ 4004.526513] Key type .llcrypt unregistered [ 4004.538278] Key type ._llcrypt unregistered [ 4041.621955] Key type ._llcrypt registered [ 4041.633942] Key type .llcrypt registered [ 4041.934173] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4043.523129] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4043.667895] alg: No test for adler32 (adler32-zlib) [ 4044.893284] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 4045.168274] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4046.928276] Key type lgssc registered [ 4048.134667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4057.607361] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4057.641371] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4059.240357] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4059.265243] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.204.13/24@tcp [ 4059.287433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4064.273930] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4069.867579] LustreError: 127477:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4069.901472] LustreError: 127477:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4074.990061] LustreError: 127476:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4075.211299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.676308] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4075.712535] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.204.13/24@tcp [ 4075.762049] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4080.520081] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4090.186707] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4097.179438] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4104.105586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4104.594254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4105.666553] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 4112.453414] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4121.367111] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4134.235789] Lustre: server umount lustre-OST0000 complete [ 4141.538155] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4141.568045] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4141.581902] Lustre: Skipped 1 previous similar message [ 4146.663067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4146.673817] Lustre: Skipped 1 previous similar message [ 4148.891853] Lustre: server umount lustre-MDT0000 complete [ 4151.783141] LustreError: 127477:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4151.842044] LustreError: 127477:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4152.800136] Lustre: 126786:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788934063/real 1788934063] req@0000000032a66132 x1875833228655744/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788934079 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4152.804670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4152.847587] Lustre: 126786:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4152.858813] Lustre: Skipped 1 previous similar message [ 4154.185835] LustreError: 127457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934081 with bad export cookie 15915051226498876644 [ 4154.189739] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4154.205821] LustreError: 127457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4154.654911] Lustre: server umount lustre-MDT0001 complete [ 4167.991462] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4171.398709] Key type lgssc unregistered [ 4171.733494] LNet: 131065:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4171.743069] LNetError: 131065:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4171.773344] LNet: Removed LNI 192.168.204.113@tcp [ 4172.920447] Key type .llcrypt unregistered [ 4172.921620] Key type ._llcrypt unregistered [ 4202.227702] Key type ._llcrypt registered [ 4202.235917] Key type .llcrypt registered [ 4202.373201] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4218.313441] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4219.573946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4219.594115] alg: No test for adler32 (adler32-zlib) [ 4220.560803] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 4220.743793] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4222.393189] Key type lgssc registered [ 4223.638063] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4265.630956] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4265.661476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4267.034832] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4267.066570] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4267.190393] Lustre: lustre-MDT0000: new disk, initializing [ 4267.306496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4267.356430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4273.482454] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4286.341743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4286.521119] Lustre: 134465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4286.641327] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4286.653653] Lustre: Skipped 1 previous similar message [ 4286.802481] Lustre: lustre-MDT0001: new disk, initializing [ 4286.941482] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4287.003957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4287.018193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4292.543696] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4302.933309] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4309.054742] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4316.454889] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4316.782657] Lustre: lustre-OST0000: new disk, initializing [ 4316.787077] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4316.796526] Lustre: 136432:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4316.884637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4323.882350] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4325.983781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4325.998487] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4326.112722] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4334.094209] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4338.785109] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4339.047314] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4344.055460] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4344.307905] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4346.861695] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4346.862350] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4346.874460] LustreError: Skipped 1 previous similar message [ 4346.878860] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4346.902622] Lustre: Skipped 1 previous similar message [ 4351.977479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4351.990385] Lustre: Skipped 1 previous similar message [ 4352.381625] Lustre: server umount lustre-OST0000 complete [ 4357.096101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4357.099127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4357.111955] Lustre: Skipped 1 previous similar message [ 4357.124755] Lustre: Skipped 2 previous similar messages [ 4362.213335] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4362.219729] Lustre: Skipped 1 previous similar message [ 4362.676556] Lustre: server umount lustre-MDT0000 complete [ 4367.370950] LustreError: 134457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934294 with bad export cookie 14220938822587024719 [ 4367.380451] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4367.389137] LustreError: 134457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4367.830805] Lustre: server umount lustre-MDT0001 complete [ 4379.235919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4379.644472] LustreError: 138648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4379.659699] LustreError: 138648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4379.716780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4384.312807] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4384.737923] LustreError: 138649:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.868963] LustreError: 138648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4394.982432] LustreError: 138649:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4395.077470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4395.341735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4399.863342] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4409.650968] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4417.050889] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4425.309588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4425.878988] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4435.795357] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4445.575597] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4450.902269] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4451.113756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4456.881256] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4457.166321] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4467.889028] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4468.050238] Lustre: lustre-OST0001: new disk, initializing [ 4468.053444] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4468.057431] Lustre: 142020:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4468.161270] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4474.526024] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4474.550309] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4474.693054] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4475.451511] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4485.666077] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4485.846676] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4491.322516] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4491.568716] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4498.804934] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff921609548800.ost_server_uuid 50 [ 4501.094653] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff921609548800.ost_server_uuid in IDLE state after 0 sec [ 4510.689464] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4510.694027] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4510.716874] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4510.717405] Lustre: Skipped 1 previous similar message [ 4510.727443] Lustre: Skipped 1 previous similar message [ 4512.733776] Lustre: server umount lustre-OST0001 complete [ 4515.831093] LustreError: 140474:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4515.853891] LustreError: 140474:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4517.346239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4517.357275] LustreError: Skipped 1 previous similar message [ 4517.368840] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4520.936743] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4520.950579] Lustre: Skipped 2 previous similar messages [ 4523.487438] Lustre: server umount lustre-OST0000 complete [ 4528.609413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4528.620941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4528.636690] Lustre: Skipped 1 previous similar message [ 4533.689280] Lustre: server umount lustre-MDT0000 complete [ 4536.292534] LustreError: 138648:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4536.309136] LustreError: 138648:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4537.614328] LustreError: 138629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934464 with bad export cookie 14220938822587025601 [ 4537.631140] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4537.635121] LustreError: 138629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4538.037208] Lustre: server umount lustre-MDT0001 complete [ 4561.567684] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4565.232421] Key type lgssc unregistered [ 4565.722209] LNet: 145025:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4565.737035] LNetError: 145025:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4565.765253] LNet: Removed LNI 192.168.204.113@tcp [ 4566.961771] Key type .llcrypt unregistered [ 4566.965176] Key type ._llcrypt unregistered [ 4614.656640] Key type ._llcrypt registered [ 4614.660151] Key type .llcrypt registered [ 4614.804163] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4632.373807] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4633.760206] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4633.787232] alg: No test for adler32 (adler32-zlib) [ 4634.832513] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 4635.052993] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4636.704236] Key type lgssc registered [ 4638.065619] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4680.103283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4680.146964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4681.447632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4681.489312] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4681.571459] Lustre: lustre-MDT0000: new disk, initializing [ 4681.651586] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4681.674464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4687.278490] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4698.741450] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4698.853327] Lustre: 149181:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4698.962621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4698.974356] Lustre: Skipped 1 previous similar message [ 4699.074401] Lustre: lustre-MDT0001: new disk, initializing [ 4699.154798] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4699.188348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4699.199346] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4704.623128] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4715.106106] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4723.030976] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4730.761530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4731.092574] Lustre: lustre-OST0000: new disk, initializing [ 4731.098398] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4731.108285] Lustre: 151150:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4731.259545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4736.629211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4736.639657] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4736.768461] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4738.981777] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4750.213615] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4755.005479] Lustre: Failing over lustre-OST0000 [ 4755.458579] Lustre: server umount lustre-OST0000 complete [ 4772.836021] Lustre: 147403:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788934683/real 1788934683] req@0000000010ffce3a x1875833846270720/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788934699 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4772.867919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4773.857038] Lustre: 147404:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788934684/real 1788934684] req@00000000319222bf x1875833846271616/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788934700 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4773.857906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4773.880604] Lustre: 147404:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4775.044493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4775.376822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4775.392791] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4778.400171] Lustre: 147404:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788934689/real 1788934689] req@00000000c4a17adf x1875833846273024/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788934705 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4778.457658] Lustre: 147404:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4782.114802] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4782.506656] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4783.521129] Lustre: 147404:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788934694/real 1788934694] req@000000007ff8acd9 x1875833846274176/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788934710 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4784.091565] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4784.091955] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4786.365431] Lustre: Failing over lustre-MDT0000 [ 4786.749315] Lustre: server umount lustre-MDT0000 complete [ 4787.617720] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4787.641699] Lustre: Skipped 1 previous similar message [ 4788.641542] Lustre: 147404:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788934699/real 1788934699] req@00000000bd9aa1c4 x1875833846275200/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788934715 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4788.676369] Lustre: 147404:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4792.811816] LustreError: 149192:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4792.845286] LustreError: 149192:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4794.278824] LustreError: 149189:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.922863] LustreError: 149192:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4797.940861] LustreError: 149192:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4803.055917] LustreError: 149187:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4803.087716] LustreError: 149187:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4804.064099] Lustre: 147402:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788934714/real 1788934714] req@00000000053314f5 x1875833846286720/t0(0) o400->MGC192.168.204.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788934730 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4804.098444] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4807.235796] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4809.189060] LustreError: 149193:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4809.200753] LustreError: 149193:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 4813.282105] LustreError: 147401:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d4bdeb86 x1875833846292224/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4813.639422] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4813.710870] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4814.762437] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4818.597369] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4818.942698] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4818.950521] Lustre: Skipped 1 previous similar message [ 4818.975190] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [ 4819.026409] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4829.158679] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4829.170830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4829.179764] Lustre: Skipped 1 previous similar message [ 4829.198526] Lustre: Skipped 1 previous similar message [ 4833.962748] Lustre: server umount lustre-OST0000 complete [ 4839.393478] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4839.404213] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4839.414193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4839.458869] Lustre: Skipped 1 previous similar message [ 4844.514715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4844.522730] Lustre: Skipped 2 previous similar messages [ 4844.953690] Lustre: server umount lustre-MDT0000 complete [ 4849.536117] LustreError: 149986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788934776 with bad export cookie 15406076676943119880 [ 4849.545622] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4849.556706] LustreError: 149986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4849.960224] Lustre: server umount lustre-MDT0001 complete [ 4862.053918] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 4865.013312] Key type lgssc unregistered [ 4865.432799] LNet: 154812:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4865.463266] LNetError: 154812:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4865.496677] LNet: Removed LNI 192.168.204.113@tcp [ 4866.379070] Key type .llcrypt unregistered [ 4866.380819] Key type ._llcrypt unregistered [ 4906.932618] Key type ._llcrypt registered [ 4906.937242] Key type .llcrypt registered [ 4907.066255] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 4908.361767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4908.560629] alg: No test for adler32 (adler32-zlib) [ 4909.728917] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 4909.971465] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 4911.664545] Key type lgssc registered [ 4912.956253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4921.366541] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4921.443640] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4923.097100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4928.475161] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 4932.582249] LustreError: 156333:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4932.613285] LustreError: 156333:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4937.698983] LustreError: 156334:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4938.848325] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4942.841191] LustreError: 156333:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4943.734060] LustreError: 156362:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4943.750859] LustreError: 156362:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4943.758748] LustreError: 156362:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4944.211423] Lustre: server umount lustre-MDT0000 complete [ 4962.455846] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 4974.671185] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5018.530955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5018.964380] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5019.032085] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5019.167953] Lustre: lustre-MDT0000: new disk, initializing [ 5019.337519] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5019.369624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5025.468662] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5038.271627] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5038.507526] Lustre: 159669:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5038.598478] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5038.605460] Lustre: Skipped 1 previous similar message [ 5038.741944] Lustre: lustre-MDT0001: new disk, initializing [ 5038.871128] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5038.937211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5038.952736] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5045.120695] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5057.210743] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5065.208427] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5073.907216] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5074.228236] Lustre: lustre-OST0000: new disk, initializing [ 5074.233783] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5074.239691] Lustre: 161637:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5074.357771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5081.197697] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5081.206788] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5081.418598] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5082.841291] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5094.853229] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5099.264340] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5101.476920] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:23:45 (1788935025) [ 5402.597517] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5402.606246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5402.623245] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5404.130512] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5404.131934] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5408.970445] Lustre: server umount lustre-MDT0000 complete [ 5409.249854] LustreError: 159680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5409.277369] LustreError: 159680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5413.566806] LustreError: 160463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935340 with bad export cookie 4802572244923138376 [ 5413.576119] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5413.577282] LustreError: 160463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5414.369603] LustreError: 159674:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5414.372800] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5414.413598] Lustre: Skipped 1 previous similar message [ 5414.422461] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5414.433906] Lustre: Skipped 2 previous similar messages [ 5418.402266] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5420.268803] Lustre: server umount lustre-MDT0001 complete [ 5431.569248] Lustre: server umount lustre-OST0000 complete [ 5445.419826] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 5458.689206] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5503.076241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5503.530850] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5503.589243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5503.819464] Lustre: lustre-MDT0000: new disk, initializing [ 5503.942590] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5503.962238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5509.392813] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5522.499356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5522.637048] Lustre: 166307:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5522.691870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5522.708909] Lustre: Skipped 1 previous similar message [ 5522.773383] Lustre: lustre-MDT0001: new disk, initializing [ 5522.867904] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5522.905312] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5522.919938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5528.668909] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5541.023583] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5548.452613] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5556.413895] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5556.698396] Lustre: lustre-OST0000: new disk, initializing [ 5556.704678] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5556.716227] Lustre: 168274:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5556.841805] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5564.444481] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5564.506054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5564.522549] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5564.679694] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5576.696121] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5582.784441] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5583.094458] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5589.656676] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5589.947456] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5595.629269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5595.635342] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5595.644370] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5595.664459] Lustre: Skipped 1 previous similar message [ 5595.690933] LustreError: Skipped 1 previous similar message [ 5599.030515] Lustre: server umount lustre-OST0000 complete [ 5604.833836] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5604.851417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5604.873239] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5604.878590] Lustre: Skipped 1 previous similar message [ 5609.888200] Lustre: server umount lustre-MDT0000 complete [ 5610.977330] LustreError: 166320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5611.008261] LustreError: 166320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5613.918605] LustreError: 166301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935540 with bad export cookie 4802572244923297213 [ 5613.932564] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5614.353986] Lustre: server umount lustre-MDT0001 complete [ 5624.798449] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:32:29 (1788935549) [ 5645.889290] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 5656.885767] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5703.015931] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5703.322202] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5703.372275] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5703.541699] Lustre: lustre-MDT0000: new disk, initializing [ 5703.715365] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5703.755313] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5709.941847] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5723.313894] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5723.514705] Lustre: 173037:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5723.946140] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5729.221436] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5741.975917] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5750.223485] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5759.246502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5759.648374] Lustre: lustre-OST0000: new disk, initializing [ 5759.653203] Lustre: Skipped 1 previous similar message [ 5759.661615] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5759.676750] Lustre: Skipped 2 previous similar messages [ 5759.686246] Lustre: 175016:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5768.362389] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5768.373475] Lustre: Skipped 1 previous similar message [ 5768.383837] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5768.522826] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5768.602169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5779.607761] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5789.154078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5789.160836] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5789.169923] Lustre: Skipped 1 previous similar message [ 5789.177701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5789.186433] Lustre: Skipped 3 previous similar messages [ 5795.044254] Lustre: server umount lustre-OST0000 complete [ 5800.931287] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5814.240505] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5814.558210] Lustre: server umount lustre-MDT0000 complete [ 5814.761723] LustreError: 173044:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5814.798597] LustreError: 173044:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5819.151156] LustreError: 173027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935746 with bad export cookie 4802572244923298095 [ 5819.159245] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5819.174368] LustreError: 173027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 5819.718713] Lustre: server umount lustre-MDT0001 complete [ 5831.232746] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:35:55 (1788935755) [ 5851.614723] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 5863.368536] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 5911.714146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5912.232316] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5912.293943] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5912.412462] Lustre: lustre-MDT0000: new disk, initializing [ 5912.536367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5912.544593] Lustre: Skipped 2 previous similar messages [ 5912.596242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5917.369719] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5928.577086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5928.799313] Lustre: 179345:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5929.170772] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5934.485271] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5945.475178] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5952.602634] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5961.693494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5962.052983] Lustre: 181323:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5968.396793] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 5968.981314] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5969.102352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5978.978875] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5989.861920] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5989.870703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5989.875753] LustreError: Skipped 1 previous similar message [ 5989.878140] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5989.878147] Lustre: Skipped 7 previous similar messages [ 5989.903552] Lustre: Skipped 3 previous similar messages [ 5993.975993] Lustre: server umount lustre-OST0000 complete [ 6005.223797] LustreError: 181331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6005.275119] LustreError: 181331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6005.448293] Lustre: server umount lustre-MDT0000 complete [ 6010.089666] LustreError: 179337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788935936 with bad export cookie 4802572244923299180 [ 6010.109688] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6010.110527] LustreError: 179337:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6010.475764] Lustre: server umount lustre-MDT0001 complete [ 6023.948057] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6026.995915] Key type lgssc unregistered [ 6027.184294] LNet: 183179:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6027.204439] LNetError: 183179:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6027.242429] LNet: Removed LNI 192.168.204.113@tcp [ 6028.162659] Key type .llcrypt unregistered [ 6028.167596] Key type ._llcrypt unregistered [ 6058.145873] Key type ._llcrypt registered [ 6058.150155] Key type .llcrypt registered [ 6058.317661] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6060.362740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6060.490564] alg: No test for adler32 (adler32-zlib) [ 6061.522431] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 6061.749893] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6063.536350] Key type lgssc registered [ 6064.845775] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6074.331381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6074.355383] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6075.957582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6080.246928] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6086.116589] LustreError: 184592:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6086.125477] LustreError: 184592:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6088.730617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6089.270653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6094.472390] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6101.710298] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6109.484260] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6117.177198] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6117.569447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6120.637155] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6124.260943] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6132.302520] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6151.651600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6151.658606] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6151.667821] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6154.722941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6154.724350] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6154.735146] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6157.863070] Lustre: server umount lustre-OST0000 complete [ 6164.963180] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6164.984337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6164.989852] Lustre: Skipped 1 previous similar message [ 6168.592646] Lustre: server umount lustre-MDT0000 complete [ 6170.090603] LustreError: 184589:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6170.110179] LustreError: 184589:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6172.548470] LustreError: 184572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936099 with bad export cookie 1945615020170148014 [ 6172.554212] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6172.556613] LustreError: 184572:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6172.834808] Lustre: server umount lustre-MDT0001 complete [ 6185.197583] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 6187.770748] Key type lgssc unregistered [ 6187.994726] LNet: 188149:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6187.998666] LNetError: 188149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6188.017788] LNet: Removed LNI 192.168.204.113@tcp [ 6189.245747] Key type .llcrypt unregistered [ 6189.252056] Key type ._llcrypt unregistered [ 6220.996649] Key type ._llcrypt registered [ 6220.999768] Key type .llcrypt registered [ 6221.160470] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6222.431823] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6222.451737] alg: No test for adler32 (adler32-zlib) [ 6223.610948] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 6223.772966] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 6225.392904] Key type lgssc registered [ 6227.186765] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6237.135026] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6237.167303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.718979] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6243.856365] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6248.932134] LustreError: 189566:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6248.951807] LustreError: 189566:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6254.051255] LustreError: 189567:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6254.277173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6254.900989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6260.259829] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6269.504538] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6275.984460] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6283.254675] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6283.784138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6288.893727] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6291.156384] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6299.924979] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6307.786740] Lustre: lustre-OST0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 6309.345571] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6309.356328] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.372783] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6310.376674] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6310.378130] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6310.392434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6312.891974] Lustre: lustre-OST0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 6312.901791] Lustre: Skipped 1 previous similar message [ 6318.000749] Lustre: lustre-OST0000: Not available for connect from 192.168.204.13@tcp (stopping) [ 6318.021346] Lustre: Skipped 2 previous similar messages [ 6321.120146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6321.550110] Lustre: server umount lustre-OST0000 complete [ 6329.670404] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff92161835d000.ost_server_uuid 50 [ 6331.672596] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6346.211091] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6346.227846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6346.231251] Lustre: Skipped 1 previous similar message [ 6346.249670] Lustre: Skipped 3 previous similar messages [ 6351.968260] Lustre: server umount lustre-MDT0000 complete [ 6355.426883] LustreError: 189566:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6355.444144] LustreError: 189566:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6356.739148] LustreError: 192147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936283 with bad export cookie 5249815190619813436 [ 6356.750068] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6356.761500] LustreError: 192147:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6357.296731] Lustre: server umount lustre-MDT0001 complete [ 6368.187282] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:44:51 (1788936291) [ 6377.066561] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6377.708326] LustreError: 193489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6377.814162] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6383.096572] LustreError: 193490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6383.738837] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6392.290450] LustreError: 193489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6392.315637] LustreError: 193489:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6393.087074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6393.388869] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6398.588901] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6408.337153] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6415.570334] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6424.195741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6424.471787] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6431.400181] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6439.087167] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6445.512354] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6445.767621] Lustre: lustre-OST0001: new disk, initializing [ 6445.770995] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6445.776485] Lustre: 196331:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6445.859073] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6447.933274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6451.572476] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6454.379379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6454.390152] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6454.514767] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6461.290284] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6464.482806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6464.494975] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6464.511578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6464.519705] Lustre: Skipped 2 previous similar messages [ 6470.573803] Lustre: server umount lustre-OST0000 complete [ 6472.120389] LustreError: 195313:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6476.373949] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6476.632099] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6480.904174] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6481.074447] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6500.844518] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6500.848077] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6500.869950] LustreError: Skipped 1 previous similar message [ 6500.875098] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6500.886360] Lustre: Skipped 2 previous similar messages [ 6500.890840] Lustre: Skipped 4 previous similar messages [ 6502.482818] Lustre: server umount lustre-OST0001 complete [ 6511.075953] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6511.093428] Lustre: Skipped 1 previous similar message [ 6512.380233] Lustre: server umount lustre-MDT0000 complete [ 6516.196375] LustreError: 193489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6516.214054] LustreError: 193489:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 6516.729651] LustreError: 193469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936443 with bad export cookie 5249815190619814507 [ 6516.738662] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6516.740685] LustreError: 193469:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6517.151534] Lustre: server umount lustre-MDT0001 complete [ 6547.514941] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:47:51 (1788936471) [ 6555.384996] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6555.730216] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6555.796283] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6556.158482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6560.809338] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6570.366332] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6570.430010] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6570.469441] Lustre: 199771:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6575.922939] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6585.288798] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6592.258135] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6600.837476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6601.075843] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6609.272857] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6610.064516] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6619.228556] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6626.402899] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6626.513404] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6626.825971] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6626.835930] Lustre: Skipped 2 previous similar messages [ 6633.805317] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6635.449839] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6644.927575] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6649.349460] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6649.365961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6649.368684] Lustre: Skipped 1 previous similar message [ 6649.396123] Lustre: Skipped 4 previous similar messages [ 6654.750200] Lustre: server umount lustre-OST0000 complete [ 6655.947713] LustreError: 201624:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6674.913314] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6674.931730] LustreError: Skipped 1 previous similar message [ 6677.290988] Lustre: server umount lustre-OST0001 complete [ 6683.617443] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6683.633050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6683.663788] Lustre: Skipped 2 previous similar messages [ 6690.049699] Lustre: server umount lustre-MDT0000 complete [ 6694.569974] LustreError: 199762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936621 with bad export cookie 5249815190619815914 [ 6694.572951] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6694.580995] LustreError: 199762:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 6694.978772] Lustre: server umount lustre-MDT0001 complete [ 6720.703360] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:50:44 (1788936644) [ 6742.276758] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 6753.876613] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 6795.919326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6796.159606] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6796.191378] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6796.284719] Lustre: lustre-MDT0000: new disk, initializing [ 6796.353299] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6796.370453] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6801.735790] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6812.826642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6812.961784] Lustre: 207889:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6813.044392] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6813.050886] Lustre: Skipped 1 previous similar message [ 6813.222763] Lustre: lustre-MDT0001: new disk, initializing [ 6813.374305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6813.394669] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6818.953690] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6830.529211] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6837.859423] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6845.263969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6845.682520] Lustre: lustre-OST0000: new disk, initializing [ 6845.686166] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6845.701749] Lustre: 209859:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6851.706047] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6851.726147] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6851.955214] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6852.331871] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6864.640391] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6869.290506] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6869.469477] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6873.800692] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6874.001492] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6877.679893] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6877.684263] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6877.694254] Lustre: Skipped 1 previous similar message [ 6877.696420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6877.722880] Lustre: Skipped 8 previous similar messages [ 6882.281361] Lustre: server umount lustre-OST0000 complete [ 6894.965284] Lustre: server umount lustre-MDT0000 complete [ 6898.149204] LustreError: 207899:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6898.162960] LustreError: 207899:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [ 6899.795875] LustreError: 208684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788936826 with bad export cookie 5249815190619817699 [ 6899.802170] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6899.808233] LustreError: 208684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6900.127274] Lustre: server umount lustre-MDT0001 complete [ 6910.089304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6916.825949] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6928.822616] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6935.728913] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6944.907703] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6952.230213] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6959.438584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6967.906826] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 6977.395803] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6983.461564] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6983.698450] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6989.323410] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6989.580357] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6995.949601] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6995.970860] LustreError: Skipped 2 previous similar messages [ 6997.752754] Lustre: server umount lustre-OST0000 complete [ 7003.979060] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7009.478369] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7042.278103] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7050.552765] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7060.826981] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7061.107613] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7061.137962] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7061.162757] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7061.179347] Lustre: Skipped 1 previous similar message [ 7063.142991] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7071.718605] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7071.729084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7071.740082] Lustre: Skipped 5 previous similar messages [ 7075.023556] Lustre: server umount lustre-OST0000 complete [ 7085.768946] Lustre: server umount lustre-MDT0000 complete [ 7091.635174] LustreError: 212058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937018 with bad export cookie 5249815190619818581 [ 7091.638845] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7091.657348] LustreError: 212058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7092.324490] Lustre: server umount lustre-MDT0001 complete [ 7107.771469] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:57:11 (1788937031) [ 7115.714494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7116.646300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7116.650207] Lustre: Skipped 6 previous similar messages [ 7121.719490] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7134.691142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7140.154388] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7149.765383] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7157.336908] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7166.863814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7174.984617] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7185.022394] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7191.170562] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7191.476297] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7197.406034] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7197.634157] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7206.371149] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7206.750423] Lustre: lustre-OST0001: new disk, initializing [ 7206.754512] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7206.761464] Lustre: 221418:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7213.220461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7213.236335] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7213.354771] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7214.243043] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7225.008685] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7231.096698] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7231.389698] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7237.918754] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7238.175734] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7244.293578] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7244.307166] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7244.316669] LustreError: Skipped 2 previous similar messages [ 7244.333647] Lustre: Skipped 10 previous similar messages [ 7247.270973] Lustre: server umount lustre-OST0001 complete [ 7249.406672] LustreError: 219872:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7249.438492] LustreError: 219872:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 13 previous similar messages [ 7256.251603] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7256.487407] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7263.135041] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7263.403743] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7294.746864] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7300.096432] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7300.121933] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7300.122706] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7302.047565] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7311.415539] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7313.838233] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7331.303505] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7331.622797] Lustre: server umount lustre-OST0001 complete [ 7353.825458] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7353.876685] Lustre: Skipped 7 previous similar messages [ 7366.112186] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7366.371771] Lustre: server umount lustre-OST0000 complete [ 7378.435523] Lustre: server umount lustre-MDT0000 complete [ 7382.758486] LustreError: 218783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937309 with bad export cookie 5249815190619819540 [ 7382.766015] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7382.775321] LustreError: 218783:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7397.348770] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7398.018491] Lustre: server umount lustre-MDT0001 complete [ 7433.597196] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 03:02:36 (1788937356) [ 7441.846767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7442.398083] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7442.438124] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7447.081495] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7459.329518] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.515741] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7459.567476] Lustre: 226687:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7466.172849] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7477.979064] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7486.022710] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7493.701991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7493.828167] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7501.275412] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7510.886489] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7520.539143] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7520.672096] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7528.645325] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7538.584675] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7541.158074] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7547.246196] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7547.518099] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7552.811205] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7553.070408] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7560.284326] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9216105b9800.ost_server_uuid 50 [ 7562.745510] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9216105b9800.ost_server_uuid in IDLE state after 0 sec [ 7563.903887] Lustre: Permanently deactivating lustre-OST0001 [ 7563.917890] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7568.681387] Lustre: Permanently reactivating lustre-OST0001 [ 7568.726943] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7568.745660] Lustre: Skipped 2 previous similar messages [ 7579.622702] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7579.628871] LustreError: Skipped 5 previous similar messages [ 7581.355225] Lustre: server umount lustre-OST0001 complete [ 7591.751936] Lustre: server umount lustre-OST0000 complete [ 7602.634810] Lustre: server umount lustre-MDT0000 complete [ 7606.566823] LustreError: 226679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937533 with bad export cookie 5249815190619820835 [ 7606.567391] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7606.579040] LustreError: 226679:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7606.759617] Lustre: server umount lustre-MDT0001 complete [ 7634.250126] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 03:05:57 (1788937557) [ 7643.082771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7643.639565] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7643.700749] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7643.713381] Lustre: Skipped 2 previous similar messages [ 7643.987495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7644.002891] Lustre: Skipped 8 previous similar messages [ 7650.676454] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7661.512410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7661.592974] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7661.617412] Lustre: 233388:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7666.204167] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7674.968606] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7683.168919] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7692.265308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7692.511148] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7692.527124] Lustre: Permanently deactivating lustre-OST0000 [ 7692.539537] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7702.038501] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7702.038972] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7709.610917] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7716.992781] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7717.052840] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7723.678632] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7724.883619] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7733.672595] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7736.315915] Lustre: Permanently reactivating lustre-OST0000 [ 7736.339750] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7736.349029] Lustre: Skipped 2 previous similar messages [ 7744.078742] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7744.079663] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 42 seconds. I think it's dead, and I am evicting it. exp 0000000084d01eb1, cur 1788937670 deadline 1788937658 last 1788937628 [ 7744.090729] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7744.096766] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7744.096776] Lustre: Skipped 1 previous similar message [ 7744.141599] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7759.330661] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7759.342607] Lustre: Skipped 24 previous similar messages [ 7760.725846] Lustre: server umount lustre-OST0001 complete [ 7764.962065] LustreError: 235238:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7764.982860] LustreError: 235238:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 43 previous similar messages [ 7779.297542] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7779.550197] Lustre: server umount lustre-OST0000 complete [ 7790.435962] Lustre: server umount lustre-MDT0000 complete [ 7794.986205] LustreError: 233379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937721 with bad export cookie 5249815190619822508 [ 7794.991485] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7794.999740] LustreError: 233379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7795.016642] LustreError: 188876:0:(import.c:762:ptlrpc_connect_import_locked()) already connecting [ 7795.774615] Lustre: server umount lustre-MDT0001 complete [ 7812.005307] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 7815.254328] Key type lgssc unregistered [ 7815.585672] LNet: 238271:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7815.594459] LNetError: 238271:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7815.624897] LNet: Removed LNI 192.168.204.113@tcp [ 7816.730074] Key type .llcrypt unregistered [ 7816.731713] Key type ._llcrypt unregistered [ 7846.361480] Key type ._llcrypt registered [ 7846.363477] Key type .llcrypt registered [ 7846.479929] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 7847.842588] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7847.968125] alg: No test for adler32 (adler32-zlib) [ 7849.192833] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 7849.568829] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 7851.352362] Key type lgssc registered [ 7853.350701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7866.232225] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7866.247541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7867.785986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7873.038853] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7878.118685] LustreError: 239736:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7878.134084] LustreError: 239736:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7883.142476] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7883.235601] LustreError: 239735:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7883.421299] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7883.470335] Lustre: 239725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7883.498470] Lustre: Permanently deactivating lustre-MDT0001 [ 7883.537912] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7883.825892] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7885.108943] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7885.115777] LustreError: 239766:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7885.124771] LustreError: 239766:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7885.141091] LustreError: 239766:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 18, retries 1, failed: rc = -108 [ 7888.852079] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7898.859260] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7906.592649] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7916.693150] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7917.116643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7925.016916] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7928.370165] LustreError: 241580:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7928.410730] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7933.415577] LustreError: 242054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7936.091705] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7937.526522] LustreError: 242054:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7937.560823] LustreError: 242054:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7937.561258] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7946.159336] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7946.672943] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7951.888327] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7951.892129] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7955.921574] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 7964.945933] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7969.681835] Lustre: Permanently reactivating lustre-MDT0001 [ 7969.695447] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7969.702267] Lustre: Skipped 2 previous similar messages [ 7971.867220] Lustre: lustre-MDT0001: Client 4c3f7388-ca99-45bc-9a95-866ddbe3e3c0 (at 192.168.204.13@tcp) reconnecting [ 7978.256552] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7980.141852] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7986.204201] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7986.510483] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7991.937654] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7992.156549] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7994.959035] Lustre: Permanently deactivating lustre-MDT0001 [ 7994.981735] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7994.989838] Lustre: Skipped 2 previous similar messages [ 8004.123126] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 8016.865332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8016.877861] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8016.907533] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8017.289826] Lustre: server umount lustre-MDT0000 complete [ 8018.413459] LustreError: 241250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8018.418713] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8018.444461] LustreError: 241250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 8018.487462] Lustre: Skipped 2 previous similar messages [ 8021.427230] LustreError: 239717:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788937948 with bad export cookie 12536574349815836262 [ 8021.445245] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8021.798952] Lustre: server umount lustre-MDT0001 complete [ 8027.886533] Lustre: server umount lustre-OST0000 complete [ 8040.629057] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:12:44 (1788937964) [ 8053.028469] Lustre: server umount lustre-OST0001 complete [ 8061.341609] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 8073.439977] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 8115.668900] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8116.097371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8116.105741] Lustre: Skipped 2 previous similar messages [ 8116.167568] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8116.292884] Lustre: lustre-MDT0000: new disk, initializing [ 8116.460151] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8116.517737] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8121.736782] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8133.708786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8133.863340] Lustre: 248052:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8133.939292] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8133.949019] Lustre: Skipped 1 previous similar message [ 8134.035296] Lustre: lustre-MDT0001: new disk, initializing [ 8134.144312] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8134.190292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8134.208897] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8138.912512] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8149.109568] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8155.884109] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8163.283738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8163.618979] Lustre: lustre-OST0000: new disk, initializing [ 8163.623670] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8163.636567] Lustre: 250023:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8163.772712] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8170.118391] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8173.650435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8173.665531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8173.805522] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8181.804298] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8184.688889] LustreError: 250031:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8194.173544] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8194.365977] Lustre: lustre-OST0001: new disk, initializing [ 8194.369079] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8194.373909] Lustre: 251139:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8194.757192] LustreError: 250031:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8194.801809] LustreError: 248060:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8200.816402] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8200.830599] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8200.952024] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8201.985996] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8204.881142] LustreError: 248060:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8212.540500] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8214.434388] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8221.665807] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8221.668873] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8221.674312] LustreError: Skipped 1 previous similar message [ 8221.682784] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8221.688503] Lustre: Skipped 1 previous similar message [ 8221.705191] Lustre: Skipped 1 previous similar message [ 8222.862817] Lustre: server umount lustre-OST0001 complete [ 8226.797339] LustreError: 250007:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8226.834712] LustreError: 250007:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8240.612072] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8240.618637] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8240.626685] LustreError: Skipped 1 previous similar message [ 8240.644944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8240.650266] Lustre: Skipped 1 previous similar message [ 8246.742864] Lustre: server umount lustre-OST0000 complete [ 8251.874499] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8251.880376] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8251.893519] Lustre: Skipped 1 previous similar message [ 8251.901182] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8251.905059] Lustre: Skipped 2 previous similar messages [ 8257.662639] Lustre: server umount lustre-MDT0000 complete [ 8261.568085] LustreError: 248046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938188 with bad export cookie 12536574349815839076 [ 8261.577241] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8261.581385] LustreError: 248046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8268.038313] Lustre: server umount lustre-MDT0001 complete [ 8279.681399] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 8282.277891] Key type lgssc unregistered [ 8282.498673] LNet: 253170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8282.512567] LNetError: 253170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8282.527851] LNet: Removed LNI 192.168.204.113@tcp [ 8283.347242] Key type .llcrypt unregistered [ 8283.349176] Key type ._llcrypt unregistered [ 8328.007624] Key type ._llcrypt registered [ 8328.013638] Key type .llcrypt registered [ 8328.143523] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 8329.396201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8329.496898] alg: No test for adler32 (adler32-zlib) [ 8330.681480] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 8330.906603] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 8332.560346] Key type lgssc registered [ 8334.648057] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8345.790229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8345.822290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.264634] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8347.307553] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8347.631978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8353.269054] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8364.281296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8364.499939] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8364.558111] Lustre: 255380:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8364.856611] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8370.058502] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8379.195247] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8386.016187] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8393.260155] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8393.506226] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8393.767064] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8400.754429] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8402.560517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8408.232406] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8462.315397] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8462.330871] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8466.335117] Lustre: server umount lustre-OST0000 complete [ 8471.396716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8488.803854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8489.125624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8495.799361] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8502.448810] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8502.460492] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8502.460531] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8505.391154] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8548.833062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8548.839454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8548.843799] LustreError: Skipped 1 previous similar message [ 8548.844784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8548.868100] Lustre: Skipped 2 previous similar messages [ 8548.886835] Lustre: Skipped 2 previous similar messages [ 8550.655301] Lustre: server umount lustre-OST0000 complete [ 8559.076226] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8559.082167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8559.091350] Lustre: Skipped 1 previous similar message [ 8564.193323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8564.210504] Lustre: Skipped 2 previous similar messages [ 8568.291377] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8568.299200] Lustre: Skipped 1 previous similar message [ 8569.312272] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8569.548378] Lustre: server umount lustre-MDT0000 complete [ 8574.084581] LustreError: 255371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938500 with bad export cookie 4403653824226054632 [ 8574.094158] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8574.106944] LustreError: 255371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8574.756358] Lustre: server umount lustre-MDT0001 complete [ 8587.496882] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 8590.665300] Key type lgssc unregistered [ 8590.918436] LNet: 260801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8590.927784] LNetError: 260801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8590.951164] LNet: Removed LNI 192.168.204.113@tcp [ 8591.824441] Key type .llcrypt unregistered [ 8591.829165] Key type ._llcrypt unregistered [ 8616.731206] Key type ._llcrypt registered [ 8616.733280] Key type .llcrypt registered [ 8616.855447] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 8617.948508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8618.078290] alg: No test for adler32 (adler32-zlib) [ 8619.208506] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 8619.519224] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 8621.249466] Key type lgssc registered [ 8622.311630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8630.275991] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8630.303554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8631.796304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8636.504640] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8642.028235] LustreError: 262215:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8642.048675] LustreError: 262215:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8645.149680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8645.510933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8650.091466] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8657.568730] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8663.049469] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8669.479181] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8669.845154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8675.524050] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8677.066121] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8683.198376] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8712.163231] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8712.171446] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8712.190236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8713.188373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8713.204272] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8713.220641] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8714.983443] Lustre: server umount lustre-OST0000 complete [ 8722.402840] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8722.420794] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8722.440585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8722.443781] Lustre: Skipped 1 previous similar message [ 8726.291237] Lustre: server umount lustre-MDT0000 complete [ 8728.555589] LustreError: 262209:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8728.599560] LustreError: 262209:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8731.894712] LustreError: 262194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938658 with bad export cookie 14121275214531881111 [ 8731.898563] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8731.906568] LustreError: 262194:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8732.291423] Lustre: server umount lustre-MDT0001 complete [ 8745.395213] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 8748.042639] Key type lgssc unregistered [ 8748.466300] LNet: 266522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8748.477164] LNetError: 266522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8748.487175] LNet: Removed LNI 192.168.204.113@tcp [ 8749.828944] Key type .llcrypt unregistered [ 8749.833170] Key type ._llcrypt unregistered [ 8768.963606] Key type ._llcrypt registered [ 8768.965241] Key type .llcrypt registered [ 8769.121406] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 8770.553954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8770.604606] alg: No test for adler32 (adler32-zlib) [ 8771.695992] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 8771.910274] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 8773.576270] Key type lgssc registered [ 8775.042481] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8784.195483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8784.249219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8785.915304] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8791.282969] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8796.132736] LustreError: 267740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8796.161548] LustreError: 267740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8800.453893] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8800.834815] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8806.210781] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8814.613840] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8821.123995] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8829.054190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8829.558453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8831.595731] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8836.949703] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8846.564725] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8861.665858] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8861.679263] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8861.694065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8861.703113] Lustre: Skipped 1 previous similar message [ 8861.725102] Lustre: Skipped 1 previous similar message [ 8866.390613] Lustre: server umount lustre-OST0000 complete [ 8871.912160] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8871.919404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8871.930177] Lustre: Skipped 1 previous similar message [ 8871.940068] Lustre: Skipped 1 previous similar message [ 8877.033134] LustreError: 267740:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8877.060547] LustreError: 267740:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8877.150348] Lustre: server umount lustre-MDT0000 complete [ 8881.533923] LustreError: 267720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938808 with bad export cookie 18161990558510827881 [ 8881.563704] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8881.947848] Lustre: server umount lustre-MDT0001 complete [ 8895.015694] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 8898.560407] Key type lgssc unregistered [ 8898.948284] LNet: 271416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8898.981099] LNetError: 271416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8899.028392] LNet: Removed LNI 192.168.204.113@tcp [ 8900.207008] Key type .llcrypt unregistered [ 8900.209218] Key type ._llcrypt unregistered [ 8929.650763] Key type ._llcrypt registered [ 8929.653330] Key type .llcrypt registered [ 8929.842517] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 8931.312849] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8931.350308] alg: No test for adler32 (adler32-zlib) [ 8932.714839] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 8933.211667] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 8934.930218] Key type lgssc registered [ 8936.186941] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8946.121830] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8946.145903] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8947.877830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8954.270373] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8957.426493] LustreError: 272831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8957.466844] LustreError: 272831:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8962.547040] LustreError: 272832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8965.003767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8965.537354] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8971.367617] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 8981.141254] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8989.046321] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8998.092449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8998.719915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9006.913057] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9017.434341] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9018.182570] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9047.028604] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9047.037816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9047.058371] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9049.062971] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9049.067423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9049.070661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9051.919608] Lustre: server umount lustre-OST0000 complete [ 9057.764754] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9057.777197] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9057.803960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9057.814511] Lustre: Skipped 1 previous similar message [ 9063.176333] Lustre: server umount lustre-MDT0000 complete [ 9064.421296] LustreError: 272832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9064.445019] LustreError: 272832:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9067.988294] LustreError: 272813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788938994 with bad export cookie 15679529231770269307 [ 9067.990345] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9067.998851] LustreError: 272813:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9068.432163] Lustre: server umount lustre-MDT0001 complete [ 9082.571742] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 9086.061656] Key type lgssc unregistered [ 9086.413297] LNet: 277142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9086.442313] LNetError: 277142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9086.460706] LNet: Removed LNI 192.168.204.113@tcp [ 9087.592746] Key type .llcrypt unregistered [ 9087.599725] Key type ._llcrypt unregistered [ 9108.936899] Key type ._llcrypt registered [ 9108.944963] Key type .llcrypt registered [ 9109.081477] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 9110.781610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9110.839135] alg: No test for adler32 (adler32-zlib) [ 9112.076483] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 9112.337571] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 9114.080296] Key type lgssc registered [ 9115.419265] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9124.773979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9124.812150] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9126.385315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9131.905518] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9136.628593] LustreError: 278380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9136.658577] LustreError: 278380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 9141.740249] LustreError: 278382:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9142.302345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9142.821488] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9148.017646] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9156.948162] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9166.312493] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9178.756098] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9179.474532] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9186.703566] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9196.893201] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9196.927768] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9212.389027] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9212.395623] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9212.398538] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9212.447742] Lustre: Skipped 1 previous similar message [ 9217.514820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9217.528153] Lustre: Skipped 2 previous similar messages [ 9217.824905] Lustre: server umount lustre-OST0000 complete [ 9224.675604] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9224.693092] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9224.725559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9227.749345] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9227.750928] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9229.522377] Lustre: server umount lustre-MDT0000 complete [ 9232.866224] LustreError: 278378:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9232.898322] LustreError: 278378:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9233.677939] LustreError: 278346:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939160 with bad export cookie 2038766222566314621 [ 9233.692375] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9234.019663] Lustre: server umount lustre-MDT0001 complete [ 9245.904323] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 9249.047640] Key type lgssc unregistered [ 9249.436865] LNet: 282083:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9249.451227] LNetError: 282083:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9249.513160] LNet: Removed LNI 192.168.204.113@tcp [ 9250.396185] Key type .llcrypt unregistered [ 9250.401391] Key type ._llcrypt unregistered [ 9262.961501] Key type ._llcrypt registered [ 9262.967246] Key type .llcrypt registered [ 9263.076325] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9287.976211] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 9308.857500] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 9311.034362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9311.135085] alg: No test for adler32 (adler32-zlib) [ 9312.355122] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 9312.695975] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 9314.392310] Key type lgssc registered [ 9315.733974] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9360.947602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9361.013850] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9362.510226] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9362.553330] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9362.654825] Lustre: lustre-MDT0000: new disk, initializing [ 9362.734548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9362.766934] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9368.101294] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9382.423043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9382.539984] Lustre: 285647:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9382.589790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9382.592736] Lustre: Skipped 1 previous similar message [ 9382.735097] Lustre: lustre-MDT0001: new disk, initializing [ 9382.867930] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9382.933247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9382.940333] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9388.670812] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9402.201860] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9410.382181] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9418.892960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9419.315944] Lustre: lustre-OST0000: new disk, initializing [ 9419.327125] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9419.339813] Lustre: 287616:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9419.476873] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9424.988764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9425.010623] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9425.136339] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9427.456718] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9440.210947] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9446.468899] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9446.711393] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9452.291897] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9452.535838] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9456.101741] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9456.110117] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9456.116665] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9456.123713] Lustre: Skipped 1 previous similar message [ 9456.160118] LustreError: Skipped 1 previous similar message [ 9461.269087] Lustre: server umount lustre-OST0000 complete [ 9466.338904] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9466.369143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9466.376884] Lustre: Skipped 2 previous similar messages [ 9471.469062] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9471.477944] Lustre: Skipped 1 previous similar message [ 9471.858990] Lustre: server umount lustre-MDT0000 complete [ 9477.903311] LustreError: 285638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939404 with bad export cookie 6789330450109328802 [ 9477.910314] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9477.922197] LustreError: 285638:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9479.017099] Lustre: server umount lustre-MDT0001 complete [ 9492.454871] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:36:56 (1788939416) [ 9500.902845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9501.438300] LustreError: 290031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9501.453481] LustreError: 290031:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9501.524561] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9506.807427] LustreError: 290032:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9506.917613] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9511.911605] LustreError: 290031:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9517.030479] LustreError: 290032:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9517.703712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9518.333125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9524.567886] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9535.197184] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9546.262109] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9556.355719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9556.681348] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9565.431764] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9576.360580] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9592.803147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9592.810389] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9592.816527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9592.829085] Lustre: Skipped 2 previous similar messages [ 9592.847210] Lustre: Skipped 1 previous similar message [ 9597.589982] Lustre: server umount lustre-OST0000 complete [ 9605.602830] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9605.622842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9605.665814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9610.114418] Lustre: server umount lustre-MDT0000 complete [ 9612.775655] LustreError: 290959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9612.809477] LustreError: 290959:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9614.964361] LustreError: 290012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939541 with bad export cookie 6789330450109329684 [ 9614.978277] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9614.984283] LustreError: 290012:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9615.413670] Lustre: server umount lustre-MDT0001 complete [ 9629.830789] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [ 9633.461337] Key type lgssc unregistered [ 9633.861917] LNet: 293593:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9633.875500] LNetError: 293593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9633.900041] LNet: Removed LNI 192.168.204.113@tcp [ 9635.326986] Key type .llcrypt unregistered [ 9635.328841] Key type ._llcrypt unregistered [ 9669.018090] Key type ._llcrypt registered [ 9669.019900] Key type .llcrypt registered [ 9669.148476] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 9685.996907] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 9687.214387] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9687.266559] alg: No test for adler32 (adler32-zlib) [ 9688.472807] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [ 9688.778182] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [ 9690.480279] Key type lgssc registered [ 9692.005712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9745.898444] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9745.928195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9747.331461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9747.382777] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9747.486373] Lustre: lustre-MDT0000: new disk, initializing [ 9747.554341] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9747.569489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9752.330389] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9763.651280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9763.738461] Lustre: 297419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9763.797284] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9763.799939] Lustre: Skipped 1 previous similar message [ 9763.852280] Lustre: lustre-MDT0001: new disk, initializing [ 9763.919359] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9763.957652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9763.976575] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9768.386254] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9779.864131] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9787.566343] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9797.272133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9797.668414] Lustre: lustre-OST2710: new disk, initializing [ 9797.697243] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9797.705470] Lustre: 299386:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9797.833223] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9804.867127] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9804.911557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9804.930421] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9805.125279] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9815.302841] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9824.238714] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9824.577098] Lustre: lustre-OST03e8: new disk, initializing [ 9824.580992] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9824.585921] Lustre: 300450:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9824.684701] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9832.663396] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9832.694055] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9832.956171] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9834.281529] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [ 9846.405692] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9855.768107] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9855.956955] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9861.585779] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9861.778758] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9868.099314] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9868.359722] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9874.265859] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9874.619655] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9880.586321] Lustre: server umount lustre-MDT0000 complete [ 9881.576292] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9881.594945] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9884.139167] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9884.154839] Lustre: Skipped 2 previous similar messages [ 9885.423546] LustreError: 300449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788939812 with bad export cookie 3866103012163367000 [ 9885.426507] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9885.448893] LustreError: 300449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9886.037459] Lustre: server umount lustre-MDT0001 complete [ 9902.187435] Lustre: server umount lustre-OST2710 complete [ 9906.149443] Lustre: 295216:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788939816/real 1788939816] req@000000008a8877ad x1875839145813888/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788939832 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9906.185548] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9907.104090] Lustre: 295218:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788939817/real 1788939817] req@00000000a7de1700 x1875839145814144/t0(0) o400->lustre-MDT0001-lwp-OST03e8@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788939833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9908.319770] Lustre: server umount lustre-OST03e8 complete [ 9934.324415] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [ 9950.634387] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [ 9989.859801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9990.129200] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9990.160201] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9990.208421] Lustre: lustre-MDT0000: new disk, initializing [ 9990.304275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9990.327500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9994.171271] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10006.681401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10006.794141] Lustre: 305728:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10006.833027] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10006.839068] Lustre: Skipped 1 previous similar message [10006.941753] Lustre: lustre-MDT0001: new disk, initializing [10007.057397] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10007.106433] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10007.129726] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10011.460888] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10021.338250] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10026.506928] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10032.436053] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10032.593370] Lustre: lustre-OST0000: new disk, initializing [10032.596476] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10032.600729] Lustre: 307693:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10032.680424] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10039.366409] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10042.420469] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10042.431966] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10042.582799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10050.981269] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10056.672817] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10056.895902] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10063.166542] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10063.527690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10068.454927] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10068.461377] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10068.478022] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10068.489242] Lustre: Skipped 1 previous similar message [10072.495217] Lustre: server umount lustre-OST0000 complete [10078.690059] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10078.695722] Lustre: Skipped 1 previous similar message [10078.699502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10078.700178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10078.707125] LustreError: Skipped 1 previous similar message [10078.711696] Lustre: Skipped 1 previous similar message [10083.701697] Lustre: server umount lustre-MDT0000 complete [10083.809540] LustreError: 305738:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10083.836783] LustreError: 305738:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [10088.012922] LustreError: 305721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940014 with bad export cookie 3866103012163368680 [10088.015653] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10088.025641] LustreError: 305721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10088.390771] Lustre: server umount lustre-MDT0001 complete [10099.221042] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:47:03 (1788940023) [10100.837066] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [10102.930622] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:47:07 (1788940027) [10107.158234] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [10134.850525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10135.261776] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10135.277159] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10135.376954] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10139.128134] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10149.840473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10150.041801] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10150.072865] Lustre: 311553:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10156.742764] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10183.231382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10183.721873] Lustre: Denying initial registration attempt from nid 192.168.204.113@tcp, specified as failover [10183.744084] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10183.758576] LustreError: 313005:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [10183.788405] LustreError: 313005:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10183.805795] LustreError: 313005:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10183.996173] Lustre: server umount lustre-OST0000 complete [10183.999457] LustreError: 313005:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10193.890898] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10193.902202] Lustre: Skipped 1 previous similar message [10193.906823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10193.910730] Lustre: Skipped 1 previous similar message [10199.009962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10199.015711] Lustre: Skipped 1 previous similar message [10201.675182] Lustre: server umount lustre-MDT0000 complete [10204.135891] LustreError: 311565:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10204.154163] LustreError: 311565:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10205.817522] LustreError: 311545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940132 with bad export cookie 3866103012163369562 [10205.827382] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10205.835537] LustreError: 311545:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10206.177467] Lustre: server umount lustre-MDT0001 complete [10220.499598] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [10225.621862] Key type lgssc unregistered [10226.343072] LNet: 314127:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10226.358569] LNetError: 314127:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10226.398207] LNet: Removed LNI 192.168.204.113@tcp [10228.166341] Key type .llcrypt unregistered [10228.168113] Key type ._llcrypt unregistered [10244.963863] Key type ._llcrypt registered [10244.966672] Key type .llcrypt registered [10245.045454] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [10246.123692] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10246.162876] alg: No test for adler32 (adler32-zlib) [10247.328934] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [10247.729850] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [10249.536226] Key type lgssc registered [10250.619128] Lustre: Echo OBD driver; http://www.lustre.org/ [10307.572713] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [10319.511757] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10319.558138] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10321.152575] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10321.218712] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10321.393550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10327.855902] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10339.226696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10339.422989] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10339.479612] Lustre: 316896:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10339.879111] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10345.171887] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10353.302665] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10365.424636] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10389.204853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10389.519822] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10389.795859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10396.713575] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10404.883643] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10409.955926] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10409.956301] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10409.965868] LustreError: Skipped 1 previous similar message [10409.969461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10415.077834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10415.092596] Lustre: Skipped 2 previous similar messages [10420.202744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10420.212952] Lustre: Skipped 1 previous similar message [10422.240161] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10422.607898] Lustre: server umount lustre-OST0000 complete [10430.436997] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10430.461607] Lustre: Skipped 1 previous similar message [10430.470967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10434.589849] Lustre: server umount lustre-MDT0000 complete [10439.309547] LustreError: 316887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940366 with bad export cookie 1684634688225970421 [10439.315046] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10439.328915] LustreError: 316887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10439.549894] Lustre: server umount lustre-MDT0001 complete [10452.884844] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [10456.130243] Key type lgssc unregistered [10456.517903] LNet: 320471:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10456.526896] LNetError: 320471:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10456.540806] LNet: Removed LNI 192.168.204.113@tcp [10457.611982] Key type .llcrypt unregistered [10457.615121] Key type ._llcrypt unregistered [10491.041136] Key type ._llcrypt registered [10491.044603] Key type .llcrypt registered [10491.214519] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [10492.259114] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10492.312884] alg: No test for adler32 (adler32-zlib) [10493.933364] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [10494.422688] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [10496.288503] Key type lgssc registered [10498.013243] Lustre: Echo OBD driver; http://www.lustre.org/ [10508.241693] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10508.280444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10509.695094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10515.491616] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10520.051223] LustreError: 321887:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10520.068365] LustreError: 321887:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10525.163486] LustreError: 321888:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10526.141550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10526.520849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10531.799886] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10541.929693] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10549.770500] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10560.428494] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10561.053215] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10569.408059] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10578.710404] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10594.754768] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.13@tcp (stopping) [10594.790344] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10594.812712] Lustre: Skipped 2 previous similar messages [10596.194252] Lustre: server umount lustre-MDT0000 complete [10599.851691] LustreError: 323395:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10600.977925] LustreError: 321868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940527 with bad export cookie 8124113814387343232 [10600.985229] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10604.986465] LustreError: 321884:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10605.017827] LustreError: 321884:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10605.031531] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10605.059608] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.13@tcp (stopping) [10605.078189] Lustre: Skipped 4 previous similar messages [10610.096697] LustreError: 321882:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10610.109536] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.13@tcp (stopping) [10610.152532] Lustre: Skipped 1 previous similar message [10615.220193] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.13@tcp (stopping) [10615.233059] Lustre: Skipped 1 previous similar message [10615.264143] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10615.808195] Lustre: server umount lustre-MDT0001 complete [10624.486637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10636.785195] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10647.388771] LustreError: 323721:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10647.408100] LustreError: 323721:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10647.515079] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10647.594770] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10653.526987] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10663.818130] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10664.477455] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10666.539592] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10669.999587] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10679.457231] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10681.621197] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10688.968346] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10690.982992] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10700.263722] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10700.276282] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10700.288530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10702.307240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10706.657545] Lustre: server umount lustre-OST0000 complete [10712.545331] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10712.549613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10712.582580] Lustre: Skipped 1 previous similar message [10712.604559] Lustre: Skipped 3 previous similar messages [10718.108761] Lustre: server umount lustre-MDT0000 complete [10722.792809] LustreError: 325505:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10722.825369] LustreError: 325505:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [10723.029030] LustreError: 321868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940649 with bad export cookie 8124113814387350722 [10723.030101] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10723.058363] LustreError: 321868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10723.717941] Lustre: server umount lustre-MDT0001 complete [10739.620254] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [10744.445788] Key type lgssc unregistered [10744.949577] LNet: 328182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10744.961843] LNetError: 328182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10744.986398] LNet: Removed LNI 192.168.204.113@tcp [10746.216484] Key type .llcrypt unregistered [10746.218842] Key type ._llcrypt unregistered [10780.272585] Key type ._llcrypt registered [10780.274040] Key type .llcrypt registered [10780.419897] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [10781.972832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10782.112075] alg: No test for adler32 (adler32-zlib) [10783.262324] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [10783.576176] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [10785.362161] Key type lgssc registered [10787.038658] Lustre: Echo OBD driver; http://www.lustre.org/ [10796.914813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10796.974740] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10798.611732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10804.402482] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10808.805183] LustreError: 329596:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10808.831237] LustreError: 329596:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10813.932596] LustreError: 329597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10815.126261] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10815.623426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10821.856630] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10831.206952] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10837.280334] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10845.665743] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10845.670362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10845.706113] Lustre: Skipped 1 previous similar message [10845.727321] Lustre: Skipped 1 previous similar message [10847.855275] Lustre: server umount lustre-MDT0000 complete [10850.785816] LustreError: 329597:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10850.818677] LustreError: 329597:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10852.095090] LustreError: 329578:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940778 with bad export cookie 5986790329775084109 [10852.116314] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10852.448087] Lustre: server umount lustre-MDT0001 complete [10871.117533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10871.517827] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10871.596377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10871.917157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10877.297302] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10887.283354] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10887.467149] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10887.532420] Lustre: 332189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10887.935783] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10893.920865] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10906.356033] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10914.851564] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10926.761191] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10927.071090] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10927.079805] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10927.090544] LustreError: 334074:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10927.097729] LustreError: 334074:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10927.105805] LustreError: 334074:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10927.231281] Lustre: server umount lustre-OST0000 complete [10927.233751] LustreError: 334074:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10940.525746] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10940.801678] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10941.104571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10947.328388] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10949.894812] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10959.151386] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10969.211183] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10969.354706] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10969.496708] Lustre: lustre-OST0001: new disk, initializing [10969.510389] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10969.522102] Lustre: 335628:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10969.635838] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10975.424482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10975.451544] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10975.560792] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10977.790490] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [10987.493091] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10996.193291] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10996.202888] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10996.210612] LustreError: Skipped 1 previous similar message [10996.214811] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10996.262701] Lustre: Skipped 1 previous similar message [10997.481770] Lustre: server umount lustre-OST0001 complete [11001.370326] LustreError: 334552:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11001.396525] LustreError: 334552:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11003.875490] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11003.875950] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11003.891622] LustreError: Skipped 1 previous similar message [11003.909749] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11003.924409] Lustre: Skipped 2 previous similar messages [11006.449170] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11006.450193] LustreError: 335636:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11006.464401] Lustre: Skipped 1 previous similar message [11006.497842] LustreError: 335636:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11011.563461] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11011.575261] Lustre: Skipped 1 previous similar message [11016.698254] LustreError: 335636:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11016.726978] LustreError: 335636:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11017.184376] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11017.386030] Lustre: server umount lustre-OST0000 complete [11026.402892] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11026.419234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11026.439633] Lustre: Skipped 1 previous similar message [11026.452573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11026.462963] Lustre: Skipped 2 previous similar messages [11028.617169] Lustre: server umount lustre-MDT0000 complete [11034.471204] LustreError: 332180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788940961 with bad export cookie 5986790329775084662 [11034.488902] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11034.496462] LustreError: 332180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [11035.553389] Lustre: server umount lustre-MDT0001 complete [11053.184336] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [11058.020271] Key type lgssc unregistered [11058.295922] LNet: 337589:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11058.303352] LNetError: 337589:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11058.332688] LNet: Removed LNI 192.168.204.113@tcp [11059.423253] Key type .llcrypt unregistered [11059.425183] Key type ._llcrypt unregistered [11151.970555] Key type ._llcrypt registered [11151.981023] Key type .llcrypt registered [11152.174210] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [11173.540836] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [11174.363770] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11174.400130] alg: No test for adler32 (adler32-zlib) [11175.437566] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [11175.675830] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [11177.360218] Key type lgssc registered [11179.179321] Lustre: Echo OBD driver; http://www.lustre.org/ [11219.019855] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11219.058794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11220.310280] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11220.355863] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11220.458288] Lustre: lustre-MDT0000: new disk, initializing [11220.589131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11220.604608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11225.916144] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11238.607381] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11238.781973] Lustre: 342524:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11238.852796] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11238.859812] Lustre: Skipped 1 previous similar message [11239.059741] Lustre: lustre-MDT0001: new disk, initializing [11239.221315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11239.297400] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11239.308364] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11245.523985] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11257.319583] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11264.589529] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11273.232559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11273.693276] Lustre: lustre-OST0000: new disk, initializing [11273.698322] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11273.706323] Lustre: 344494:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11273.810660] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11280.932825] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11281.583796] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11281.603271] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11281.697912] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11291.454359] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11296.887372] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11297.161323] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11302.770889] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11303.036629] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11307.499865] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11307.503407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11307.507618] LustreError: Skipped 1 previous similar message [11307.513781] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11307.525695] Lustre: Skipped 1 previous similar message [11307.558331] Lustre: Skipped 1 previous similar message [11311.472254] Lustre: server umount lustre-OST0000 complete [11316.194435] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11316.213681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11316.241986] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11317.729591] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11317.733203] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11322.361702] Lustre: server umount lustre-MDT0000 complete [11327.113513] LustreError: 342516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941253 with bad export cookie 8208217301604633976 [11327.129056] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11327.130414] LustreError: 342516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11327.571066] Lustre: server umount lustre-MDT0001 complete [11338.211107] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 04:07:42 (1788941262) [11346.911440] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 04:07:51 (1788941271) [11358.102453] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11358.992526] LustreError: 347249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11359.047402] LustreError: 347249:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11359.222430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11364.322348] LustreError: 347250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11364.338087] LustreError: 347250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11365.675591] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11368.418914] LustreError: 347249:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11373.548039] LustreError: 347250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11376.611846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11377.042885] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11382.975770] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11392.964771] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11400.084244] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11408.004597] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11408.365909] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11415.177510] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11423.247978] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11435.986662] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11438.995663] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11441.177719] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11443.936638] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11445.856262] Lustre: Failing over lustre-MDT0000 [11446.127859] Lustre: server umount lustre-MDT0000 complete [11448.806643] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11448.829540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11448.853065] LustreError: 347979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.555831] LustreError: 347979:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.581084] LustreError: 347979:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [11465.699111] Lustre: 340748:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788941376/real 1788941376] req@00000000422e2387 x1875840705047808/t0(0) o400->MGC192.168.204.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788941392 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11465.756138] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11466.832095] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11475.937796] LustreError: 347250:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11475.938846] LustreError: 340745:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000dc0829b1 x1875840705053824/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11475.975616] LustreError: 347250:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 16 previous similar messages [11476.311214] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11476.354122] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11480.491112] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11480.932819] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11481.601177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11481.649383] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [11481.734772] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11492.508495] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11494.402090] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11498.533320] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11500.585636] Lustre: Failing over lustre-MDT0000 [11500.832215] Lustre: server umount lustre-MDT0000 complete [11502.061250] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11502.074892] Lustre: Skipped 4 previous similar messages [11511.216756] LustreError: 347246:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11511.225017] LustreError: 347246:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [11517.230085] Lustre: 340749:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788941428/real 1788941428] req@000000007343b911 x1875840705069440/t0(0) o400->MGC192.168.204.113@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788941444 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11517.271325] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11519.772109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11527.649901] LustreError: 340745:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d8a8a182 x1875840705074176/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11527.933300] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11527.981655] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11531.735391] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11533.312022] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11533.324173] Lustre: Skipped 2 previous similar messages [11533.366177] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [11533.440872] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11534.062714] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11544.725814] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11546.334316] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11553.765161] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11553.766339] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11553.786393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11553.804589] Lustre: Skipped 3 previous similar messages [11558.415515] Lustre: server umount lustre-MDT0000 complete [11562.219196] LustreError: 351243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941489 with bad export cookie 8208217301604636692 [11562.225856] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11562.233235] LustreError: 351243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11562.694614] Lustre: server umount lustre-MDT0001 complete [11576.552958] Lustre: server umount lustre-OST0000 complete [11592.491964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11592.917408] LustreError: 353618:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11592.928139] LustreError: 353618:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 19 previous similar messages [11592.968555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11597.021828] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11605.232962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11611.148457] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11620.001027] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11627.536058] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11635.244852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11637.632858] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11642.965972] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11651.553964] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11658.215954] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11658.231422] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11658.245711] Lustre: Skipped 2 previous similar messages [11658.256472] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11664.063597] Lustre: server umount lustre-OST0000 complete [11667.428984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11667.441565] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11667.446835] Lustre: Skipped 2 previous similar messages [11673.615301] Lustre: server umount lustre-MDT0000 complete [11677.499611] LustreError: 353597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941604 with bad export cookie 8208217301604637434 [11677.516177] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11677.523500] LustreError: 353597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11678.110881] Lustre: server umount lustre-MDT0001 complete [11692.696515] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [11695.745654] Key type lgssc unregistered [11695.987933] LNet: 357152:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11695.997508] LNetError: 357152:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11696.011781] LNet: Removed LNI 192.168.204.113@tcp [11697.059343] Key type .llcrypt unregistered [11697.061882] Key type ._llcrypt unregistered [11726.309912] Key type ._llcrypt registered [11726.312570] Key type .llcrypt registered [11726.447946] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [11727.883925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11727.911344] alg: No test for adler32 (adler32-zlib) [11728.961829] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [11729.207382] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [11730.880275] Key type lgssc registered [11732.145772] Lustre: Echo OBD driver; http://www.lustre.org/ [11741.413574] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11741.435929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11742.936369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11747.154657] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11752.418969] LustreError: 358680:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11752.446702] LustreError: 358680:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11756.951020] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11757.578991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11762.930275] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11772.026629] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11778.422845] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11786.630537] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11787.020600] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11788.030840] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11793.400329] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11802.482480] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11818.980353] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11818.981614] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11818.992939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11819.003569] Lustre: Skipped 1 previous similar message [11821.656322] Lustre: server umount lustre-MDT0000 complete [11824.102915] LustreError: 358676:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11824.126853] LustreError: 358676:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11825.609206] LustreError: 358705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941752 with bad export cookie 14500000722525076152 [11825.618303] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11825.625234] LustreError: 358705:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11825.920331] Lustre: server umount lustre-MDT0001 complete [11839.427985] Lustre: server umount lustre-OST0000 complete [11854.394885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11854.900838] LustreError: 362414:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11854.983714] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11860.084511] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11860.450208] LustreError: 362415:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11864.553758] LustreError: 362414:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11869.787662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11870.206059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11874.980099] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11883.643260] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11890.042031] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11896.586305] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11896.893394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11899.954536] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11903.628904] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11911.612753] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11924.458790] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11924.466369] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11924.481267] Lustre: Skipped 3 previous similar messages [11924.485790] Lustre: Skipped 3 previous similar messages [11929.573386] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11930.152884] Lustre: server umount lustre-MDT0000 complete [11934.041625] LustreError: 362396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788941860 with bad export cookie 14500000722525077265 [11934.065164] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11934.420758] Lustre: server umount lustre-MDT0001 complete [11948.499772] Lustre: server umount lustre-OST0000 complete [11965.355194] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11966.021801] LustreError: 366323:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11966.044736] LustreError: 366323:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11966.141680] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11972.086351] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11983.147672] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11988.602185] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [11997.382819] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12003.654857] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12011.442836] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12011.914491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12011.927281] Lustre: Skipped 1 previous similar message [12018.225811] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12026.475221] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12035.446880] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [12043.747280] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12043.760627] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12043.786346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12043.791907] Lustre: Skipped 2 previous similar messages [12045.799580] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12045.804718] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12049.746454] Lustre: server umount lustre-OST0000 complete [12055.522367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12055.532918] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12055.550262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12055.562037] Lustre: Skipped 2 previous similar messages [12066.275282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12066.286726] Lustre: Skipped 5 previous similar messages [12068.320469] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12068.731738] Lustre: server umount lustre-MDT0000 complete [12070.380402] LustreError: 366320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12070.412221] LustreError: 366320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12073.234712] LustreError: 368170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942000 with bad export cookie 14500000722525078287 [12073.239794] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12073.247514] LustreError: 368170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [12073.835555] Lustre: server umount lustre-MDT0001 complete [12089.129654] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [12092.744498] Key type lgssc unregistered [12093.168741] LNet: 369886:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12093.187820] LNetError: 369886:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12093.208436] LNet: Removed LNI 192.168.204.113@tcp [12094.475336] Key type .llcrypt unregistered [12094.477403] Key type ._llcrypt unregistered [12125.348896] Key type ._llcrypt registered [12125.351096] Key type .llcrypt registered [12125.477601] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [12126.767687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12126.945130] alg: No test for adler32 (adler32-zlib) [12128.051081] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [12128.380233] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [12130.129365] Key type lgssc registered [12131.578772] Lustre: Echo OBD driver; http://www.lustre.org/ [12140.391932] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12140.420612] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12140.427774] LustreError: 371325:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [12140.447701] LustreError: 371325:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [12140.454729] LustreError: 371325:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [12140.465971] LustreError: 371325:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [12140.476063] LustreError: 371325:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12149.446300] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [12149.451911] LustreError: 371723:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [12149.465573] LustreError: 371723:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [12149.471671] LustreError: 371723:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [12149.480169] LustreError: 371723:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [12149.496802] LustreError: 371723:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [12172.962404] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [12176.256350] Key type lgssc unregistered [12176.704791] LNet: 372726:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12176.718716] LNetError: 372726:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12176.746335] LNet: Removed LNI 192.168.204.113@tcp [12177.730447] Key type .llcrypt unregistered [12177.733353] Key type ._llcrypt unregistered [12200.872526] Key type ._llcrypt registered [12200.874961] Key type .llcrypt registered [12200.983295] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [12218.080845] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [12219.816679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12219.868775] alg: No test for adler32 (adler32-zlib) [12220.973817] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [12221.191348] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [12222.920207] Key type lgssc registered [12224.372732] Lustre: Echo OBD driver; http://www.lustre.org/ [12269.755765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12269.772495] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12271.011256] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12271.051335] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12271.239583] Lustre: lustre-MDT0000: new disk, initializing [12271.357758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12271.372834] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12276.522985] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12289.228838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12289.410933] Lustre: 375879:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12289.484364] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12289.491757] Lustre: Skipped 1 previous similar message [12289.632878] Lustre: lustre-MDT0001: new disk, initializing [12289.739298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12289.788034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12289.797579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12295.702921] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12307.568910] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12314.563843] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12323.065592] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12323.377138] Lustre: lustre-OST0000: new disk, initializing [12323.384658] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12323.399204] Lustre: 377850:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12323.495816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12330.985588] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12332.647456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12332.661680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12332.810863] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12342.033561] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12347.548577] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12347.833747] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12353.486946] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12353.729698] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12358.626696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12358.644959] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12358.670880] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12363.749405] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12363.762874] Lustre: Skipped 2 previous similar messages [12368.872385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12368.877768] Lustre: Skipped 1 previous similar message [12370.400594] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12370.569817] Lustre: server umount lustre-OST0000 complete [12377.059331] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12377.070817] LustreError: Skipped 1 previous similar message [12377.081706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12377.104823] Lustre: Skipped 1 previous similar message [12377.118542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12379.105453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12384.226835] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12384.237327] Lustre: Skipped 3 previous similar messages [12389.344182] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12389.573984] Lustre: server umount lustre-MDT0000 complete [12393.899616] LustreError: 375873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942320 with bad export cookie 9012713999118253277 [12393.907735] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12393.916491] LustreError: 375873:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12394.362665] Lustre: server umount lustre-MDT0001 complete [12405.030907] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12406.844950] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 04:25:31 (1788942331) [12415.245917] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12415.833431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12420.605464] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12421.093153] LustreError: 380316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12421.128771] LustreError: 380316:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12425.203803] LustreError: 380317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12430.307416] LustreError: 380316:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12430.623115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12431.210470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12436.686282] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12445.577517] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12452.235289] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12459.543093] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12459.846729] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12466.723860] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12474.656636] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12482.520365] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12482.808325] Lustre: lustre-OST0001: new disk, initializing [12482.827901] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12482.846499] Lustre: 383151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12482.959321] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12488.864124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12488.880242] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12489.000768] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12490.283198] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12501.397600] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12508.612614] Lustre: lustre-OST0001: Not available for connect from 192.168.204.13@tcp (stopping) [12508.617567] Lustre: Skipped 2 previous similar messages [12509.671088] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12509.676733] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12509.678675] LustreError: Skipped 1 previous similar message [12509.691165] Lustre: Skipped 1 previous similar message [12511.346898] Lustre: server umount lustre-OST0001 complete [12513.710059] LustreError: 382138:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.204.13@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12519.938442] LustreError: 382140:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12519.953322] LustreError: 382140:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12530.148733] LustreError: 382138:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12530.160154] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12530.162912] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12530.162923] Lustre: Skipped 2 previous similar messages [12530.166286] LustreError: 382138:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12530.174409] Lustre: Skipped 1 previous similar message [12534.220619] Lustre: server umount lustre-OST0000 complete [12538.854986] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12538.861645] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12544.229531] Lustre: server umount lustre-MDT0000 complete [12547.642017] LustreError: 382160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942474 with bad export cookie 9012713999118254159 [12547.647014] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12547.650684] LustreError: 382160:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12554.067200] Lustre: server umount lustre-MDT0001 complete [12565.379761] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [12568.544769] Key type lgssc unregistered [12568.878944] LNet: 385138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12568.889215] LNetError: 385138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12568.910681] LNet: Removed LNI 192.168.204.113@tcp [12569.896102] Key type .llcrypt unregistered [12569.898384] Key type ._llcrypt unregistered [12625.060617] Key type ._llcrypt registered [12625.062696] Key type .llcrypt registered [12625.165477] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [12626.367785] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12626.484335] alg: No test for adler32 (adler32-zlib) [12627.602183] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [12627.911718] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [12629.648232] Key type lgssc registered [12630.854818] Lustre: Echo OBD driver; http://www.lustre.org/ [12638.942879] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12638.964267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12640.194946] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12640.214253] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12640.393968] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12645.515219] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12655.503472] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12655.691398] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12655.796487] Lustre: 387936:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12656.183849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12661.275644] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12669.332382] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12675.851171] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12678.625761] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12678.644585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12678.651054] Lustre: Skipped 1 previous similar message [12683.747478] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12683.754045] Lustre: Skipped 1 previous similar message [12684.415366] Lustre: server umount lustre-MDT0000 complete [12688.637454] LustreError: 387927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942615 with bad export cookie 18257151685913351746 [12688.645052] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12688.653717] LustreError: 387927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12689.215735] Lustre: server umount lustre-MDT0001 complete [12700.114836] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 04:30:24 (1788942624) [12707.681920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12708.246601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12712.698381] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12713.445273] LustreError: 390451:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12713.462919] LustreError: 390451:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [12718.566032] LustreError: 390450:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12721.988703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12722.432292] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12727.336215] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12734.715574] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12740.490971] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12747.226931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12747.482492] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12747.749718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12754.243178] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12762.548164] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12771.732702] Lustre: Permanently deactivating lustre-OST0000 [12771.752717] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12780.681543] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12782.985604] LustreError: 393440:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12782.995613] LustreError: 393440:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12783.005433] LustreError: 393440:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12786.321415] LustreError: 393489:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12786.328570] LustreError: 393489:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12795.076381] Lustre: server umount lustre-OST0000 complete [12798.149630] LustreError: 393738:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12798.157376] LustreError: 393738:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12801.510526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12801.520238] Lustre: Skipped 1 previous similar message [12801.523700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12806.625546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12806.631533] Lustre: Skipped 2 previous similar messages [12809.824237] Lustre: server umount lustre-MDT0000 complete [12811.749101] LustreError: 393252:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12811.760926] LustreError: 393252:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12814.452773] LustreError: 390432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942741 with bad export cookie 18257151685913352327 [12814.453790] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12814.459707] LustreError: 390432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12825.278238] Lustre: server umount lustre-MDT0001 complete [12835.977554] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12841.981434] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12846.482934] LustreError: 394898:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12853.355715] Lustre: 395237:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.204.113@tcp [12856.326683] Lustre: server umount MGS complete [12871.907030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12872.428128] LustreError: 395963:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12872.457883] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12872.466677] Lustre: Skipped 1 previous similar message [12872.552962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12877.616556] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12877.795824] LustreError: 395964:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12886.184155] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12886.464017] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12891.713401] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12899.576956] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12905.588652] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12907.647818] Lustre: Permanently reactivating lustre-OST0000 [12907.678493] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12907.685963] Lustre: Skipped 2 previous similar messages [12938.210127] LustreError: 397824:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12944.658178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12944.977795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12944.987615] Lustre: Skipped 1 previous similar message [12950.761391] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [12958.254133] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12963.813057] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12963.815295] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12963.819460] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12963.844790] Lustre: Skipped 2 previous similar messages [12969.449072] Lustre: server umount lustre-OST0000 complete [12974.066810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12974.070054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12974.088551] Lustre: Skipped 1 previous similar message [12974.100267] Lustre: Skipped 4 previous similar messages [12980.015865] Lustre: server umount lustre-MDT0000 complete [12984.070831] LustreError: 395943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788942910 with bad export cookie 18257151685913353377 [12984.088139] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12984.092621] LustreError: 395943:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12984.584930] Lustre: server umount lustre-MDT0001 complete [12995.704766] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [12998.656918] Key type lgssc unregistered [12999.041557] LNet: 400421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12999.049154] LNetError: 400421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12999.083308] LNet: Removed LNI 192.168.204.113@tcp [13000.192124] Key type .llcrypt unregistered [13000.193700] Key type ._llcrypt unregistered [13020.677803] Key type ._llcrypt registered [13020.679889] Key type .llcrypt registered [13020.803729] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [13037.439717] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [13038.796493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13038.955867] alg: No test for adler32 (adler32-zlib) [13040.064380] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [13040.310627] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [13041.993841] Key type lgssc registered [13043.148787] Lustre: Echo OBD driver; http://www.lustre.org/ [13077.631320] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13077.661242] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13078.895260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13078.934646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13079.040301] Lustre: lustre-MDT0000: new disk, initializing [13079.185095] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13079.208353] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13083.841030] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13094.989319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13095.302962] Lustre: 403576:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13095.410682] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13095.414987] Lustre: Skipped 1 previous similar message [13095.515580] Lustre: lustre-MDT0001: new disk, initializing [13095.620929] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13095.669266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13095.694238] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13100.513702] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13110.456667] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13117.681918] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13125.225571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13125.473512] Lustre: lustre-OST0000: new disk, initializing [13125.479682] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13125.486804] Lustre: 405546:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13125.584025] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13130.864917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13130.884313] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13131.061766] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13133.104505] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13142.879445] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13147.603410] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13147.811913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13152.796139] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13153.157166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13156.844484] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13156.846771] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13156.863659] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13156.878758] Lustre: Skipped 1 previous similar message [13161.578068] Lustre: server umount lustre-OST0000 complete [13167.079540] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13167.091909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13167.106400] Lustre: Skipped 2 previous similar messages [13171.970097] Lustre: server umount lustre-MDT0000 complete [13172.203328] LustreError: 404659:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13172.234468] LustreError: 404659:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13175.407129] LustreError: 403567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943102 with bad export cookie 13239963497898572070 [13175.410887] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13175.417709] LustreError: 403567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13175.868296] Lustre: server umount lustre-MDT0001 complete [13184.777129] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13186.458885] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 04:38:30 (1788943110) [13193.869105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13194.375421] LustreError: 408008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13194.455602] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13199.103291] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13199.846449] LustreError: 408009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13204.973643] LustreError: 408008:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13207.790192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13208.183514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13212.781963] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13220.571475] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13226.390906] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13232.994364] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13233.133689] LustreError: 409828:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13233.320105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13239.438202] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13246.718797] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13248.503796] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13252.071376] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13259.246073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13259.255783] LustreError: Skipped 1 previous similar message [13259.263117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13259.285800] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13259.293226] Lustre: Skipped 1 previous similar message [13260.257962] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13263.414709] Lustre: server umount lustre-OST0000 complete [13269.474775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13269.480253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13269.489093] Lustre: Skipped 1 previous similar message [13269.496289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13269.500839] Lustre: Skipped 2 previous similar messages [13275.620047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13275.635575] Lustre: Skipped 3 previous similar messages [13281.248190] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13281.426473] Lustre: server umount lustre-MDT0000 complete [13284.852583] LustreError: 407988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943211 with bad export cookie 13239963497898572952 [13284.856511] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13284.866837] LustreError: 407988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13285.117759] Lustre: server umount lustre-MDT0001 complete [13294.645721] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [13296.866851] Key type lgssc unregistered [13297.087884] LNet: 411733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13297.098519] LNetError: 411733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13297.118222] LNet: Removed LNI 192.168.204.113@tcp [13297.766487] Key type .llcrypt unregistered [13297.769529] Key type ._llcrypt unregistered [13316.431612] Key type ._llcrypt registered [13316.437428] Key type .llcrypt registered [13316.526215] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [13316.768537] Key type .llcrypt unregistered [13316.772570] Key type ._llcrypt unregistered [13331.730741] Key type ._llcrypt registered [13331.734477] Key type .llcrypt registered [13331.978865] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [13333.527208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13333.597995] alg: No test for adler32 (adler32-zlib) [13334.645518] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [13334.914484] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [13336.652448] Key type lgssc registered [13338.414875] Lustre: Echo OBD driver; http://www.lustre.org/ [13347.006655] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13347.035594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13348.784039] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13354.010742] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13359.073847] LustreError: 413928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13359.086992] LustreError: 413928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13363.103640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13363.493207] LustreError: 413929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13363.516229] LustreError: 413929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13363.525789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13368.610601] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13368.806600] LustreError: 413928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13368.831875] LustreError: 413928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13373.933717] LustreError: 413929:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13373.955608] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13373.961727] LustreError: 413929:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13376.601109] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13379.045842] LustreError: 413928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13379.059464] LustreError: 413928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13382.692029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13383.089522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13388.728924] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13398.497146] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13398.510495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13398.518506] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13398.535036] Lustre: Skipped 1 previous similar message [13403.626558] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13403.642063] Lustre: Skipped 2 previous similar messages [13404.453289] Lustre: server umount lustre-OST0000 complete [13408.739396] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13408.740730] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13408.759189] Lustre: Skipped 1 previous similar message [13413.596766] Lustre: server umount lustre-MDT0000 complete [13413.865342] LustreError: 413928:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13413.885791] LustreError: 413928:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13416.967952] LustreError: 415673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943343 with bad export cookie 9055576435674568508 [13416.969227] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13416.979918] LustreError: 415673:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13417.190033] Lustre: server umount lustre-MDT0001 complete [13425.739385] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [13428.598497] Key type lgssc unregistered [13428.956595] LNet: 417077:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13428.964238] LNetError: 417077:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13428.982601] LNet: Removed LNI 192.168.204.113@tcp [13429.943419] Key type .llcrypt unregistered [13429.946903] Key type ._llcrypt unregistered [13454.592848] Key type ._llcrypt registered [13454.595070] Key type .llcrypt registered [13454.706163] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [13456.244661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13456.332135] alg: No test for adler32 (adler32-zlib) [13457.360686] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [13457.555985] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [13459.225462] Key type lgssc registered [13460.253343] Lustre: Echo OBD driver; http://www.lustre.org/ [13468.956613] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13468.974897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13484.513136] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13489.632550] LustreError: 418464:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.204.113@tcp: failed processing log, type 4: rc = -110 [13516.320359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13522.975485] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13530.473132] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13537.289582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13537.433461] LustreError: 417798:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000047d9f29c x1875843097823488/t0(0) o250->MGC192.168.204.113@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13537.668849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13537.712125] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13541.872768] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13542.890918] LustreError: 418490:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13542.917450] LustreError: 418490:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13548.002289] LustreError: 418489:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13549.616863] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13550.016400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13553.518832] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13562.479913] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13570.459733] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13580.770768] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13580.787887] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13580.806385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13581.806141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13581.808765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13586.370745] Lustre: server umount lustre-OST0000 complete [13591.014519] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13591.018345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13591.024133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13591.026285] Lustre: Skipped 1 previous similar message [13592.851342] Lustre: server umount lustre-MDT0000 complete [13596.158382] LustreError: 418470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943523 with bad export cookie 8040550959546478462 [13596.169746] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13596.171420] LustreError: 418470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13596.470323] Lustre: server umount lustre-MDT0001 complete [13607.555620] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [13609.877974] Key type lgssc unregistered [13610.165773] LNet: 422032:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13610.177919] LNetError: 422032:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13611.238942] LNet: Removed LNI 192.168.204.113@tcp [13612.321563] Key type .llcrypt unregistered [13612.324538] Key type ._llcrypt unregistered [13640.198245] Key type ._llcrypt registered [13640.202495] Key type .llcrypt registered [13640.410717] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [13641.913255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13641.970665] alg: No test for adler32 (adler32-zlib) [13643.127485] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [13643.339653] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [13644.968242] Key type lgssc registered [13646.515910] Lustre: Echo OBD driver; http://www.lustre.org/ [13654.510106] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13654.526105] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13655.867326] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13659.752643] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13665.255899] LustreError: 423450:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13665.277824] LustreError: 423450:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13668.510505] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13668.873862] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13673.816885] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13681.091366] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13687.854151] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13696.459230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13696.918906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13702.024585] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13704.627150] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13712.684507] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13718.501445] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13718.503039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13718.527821] Lustre: Skipped 2 previous similar messages [13718.531403] Lustre: Skipped 1 previous similar message [13719.507719] Lustre: lustre-MDT0000: Not available for connect from 192.168.204.13@tcp (stopping) [13719.518215] Lustre: Skipped 1 previous similar message [13722.940731] Lustre: server umount lustre-MDT0000 complete [13723.627475] LustreError: 423444:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13723.654749] LustreError: 423444:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13725.595578] LustreError: 423430:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.204.13@tcp arrived at 1788943652 with bad export cookie 14366354563669595345 [13727.742073] LustreError: 423429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788943654 with bad export cookie 14366354563669594456 [13727.757697] LustreError: 423429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13727.758369] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13728.746996] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13728.752160] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13728.760185] LustreError: 423446:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13728.760196] LustreError: 423446:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13728.771965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13732.769154] LustreError: 423450:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13732.770539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13732.781513] LustreError: 423450:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13733.988359] Lustre: server umount lustre-OST0000 complete [13737.891980] LustreError: 423449:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13737.904785] LustreError: 423449:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13741.018583] Lustre: server umount lustre-MDT0001 complete [13751.988919] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [13755.201426] Key type lgssc unregistered [13755.596058] LNet: 427133:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13755.602969] LNetError: 427133:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13756.649306] LNet: Removed LNI 192.168.204.113@tcp [13757.571148] Key type .llcrypt unregistered [13757.576765] Key type ._llcrypt unregistered [13778.333295] Key type ._llcrypt registered [13778.334856] Key type .llcrypt registered [13778.437236] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [13779.947045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13779.975569] alg: No test for adler32 (adler32-zlib) [13781.071482] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [13781.266037] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [13782.912206] Key type lgssc registered [13784.250097] Lustre: Echo OBD driver; http://www.lustre.org/ [13792.061367] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13792.075621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13793.535601] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13798.736797] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13804.011297] LustreError: 428526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13804.031886] LustreError: 428526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13807.467817] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13808.026157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13812.813823] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13822.516995] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13830.915540] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13841.614170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13842.085855] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13849.603052] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [13858.546842] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13863.613614] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13869.033068] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13869.038996] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13869.050756] Lustre: Skipped 1 previous similar message [13869.077922] Lustre: Skipped 1 previous similar message [13872.789450] Lustre: server umount lustre-MDT0001 complete [13874.145513] LustreError: 428526:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13874.166757] LustreError: 428526:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13879.267730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13879.268455] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13879.274380] LustreError: 428525:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13879.274395] LustreError: 428525:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13879.316329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13879.322324] Lustre: Skipped 1 previous similar message [13884.387080] LustreError: 429052:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13884.388976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13885.446716] Lustre: server umount lustre-OST0000 complete [13889.332655] Lustre: server umount lustre-MDT0000 complete [13903.595312] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [13906.730565] Key type lgssc unregistered [13907.069943] LNet: 432212:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13907.078561] LNetError: 432212:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13907.121334] LNet: Removed LNI 192.168.204.113@tcp [13908.122968] Key type .llcrypt unregistered [13908.124810] Key type ._llcrypt unregistered [13932.012303] Key type ._llcrypt registered [13932.016629] Key type .llcrypt registered [13932.147463] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [13944.713536] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [13946.218541] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13946.283537] alg: No test for adler32 (adler32-zlib) [13947.486726] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [13947.771925] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [13949.530664] Key type lgssc registered [13950.668837] Lustre: Echo OBD driver; http://www.lustre.org/ [13988.508204] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [14004.169796] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14004.206510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14005.408710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14005.478173] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14005.589823] Lustre: lustre-MDT0000: new disk, initializing [14005.732929] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14005.769365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14010.525538] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [14026.770125] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14027.028853] Lustre: lustre-OST0000: new disk, initializing [14027.032886] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14027.041889] Lustre: Skipped 1 previous similar message [14027.050268] Lustre: 437225:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14027.152812] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14032.514757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14032.523908] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14032.557844] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14033.333357] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [14043.725441] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14060.303639] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14060.482812] Lustre: 436206:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14060.589706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14060.697637] Lustre: lustre-MDT0001: new disk, initializing [14060.875649] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14060.929321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14060.947756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14066.405593] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [14077.251881] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14077.539173] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14082.933713] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14083.136547] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14089.716378] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14089.744954] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14089.752514] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14089.759467] Lustre: Skipped 1 previous similar message [14092.768784] Lustre: lustre-MDT0001: Not available for connect from 192.168.204.13@tcp (stopping) [14094.820494] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14094.831074] Lustre: Skipped 1 previous similar message [14095.159343] Lustre: server umount lustre-MDT0001 complete [14105.065200] LustreError: 436212:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14105.083608] LustreError: 436212:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [14110.185765] LustreError: 436212:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14110.217357] LustreError: 436212:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14114.278827] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14114.286559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14114.299700] Lustre: Skipped 1 previous similar message [14114.308867] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14115.314454] LustreError: 436218:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14119.985798] Lustre: server umount lustre-OST0000 complete [14120.418128] LustreError: 439796:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14130.755369] Lustre: server umount lustre-MDT0000 complete [14146.175568] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [14149.067407] Key type lgssc unregistered [14149.332991] LNet: 440975:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14149.342081] LNetError: 440975:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14149.358704] LNet: Removed LNI 192.168.204.113@tcp [14150.382166] Key type .llcrypt unregistered [14150.383956] Key type ._llcrypt unregistered [14193.600719] Key type ._llcrypt registered [14193.603422] Key type .llcrypt registered [14193.740986] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [14194.081383] Key type .llcrypt unregistered [14194.083057] Key type ._llcrypt unregistered [14211.315421] Key type ._llcrypt registered [14211.318618] Key type .llcrypt registered [14211.451922] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [14212.741879] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14212.769268] alg: No test for adler32 (adler32-zlib) [14213.780792] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [14213.978418] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [14215.649321] Key type lgssc registered [14216.692959] Lustre: Echo OBD driver; http://www.lustre.org/ [14248.308787] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14248.334627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14249.577340] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14249.616057] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14249.702358] Lustre: lustre-MDT0000: new disk, initializing [14249.813443] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14249.872715] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14255.435218] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [14266.850197] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14266.978926] Lustre: 444347:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14267.069585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14267.080447] Lustre: Skipped 1 previous similar message [14267.219653] Lustre: lustre-MDT0001: new disk, initializing [14267.351383] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14267.414347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14267.429496] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14272.376941] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [14283.776559] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14289.587251] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14296.729578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14297.001319] Lustre: lustre-OST0000: new disk, initializing [14297.006299] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14297.015603] Lustre: 446316:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14297.144150] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14302.200027] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [14306.411878] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14306.425841] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14306.589228] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14312.511598] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14322.151264] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14322.160234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14322.204231] Lustre: Skipped 2 previous similar messages [14322.223351] Lustre: Skipped 2 previous similar messages [14327.273561] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14327.279579] Lustre: Skipped 2 previous similar messages [14328.150817] Lustre: server umount lustre-MDT0000 complete [14332.595900] LustreError: 444340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944259 with bad export cookie 14886028289535222371 [14332.600567] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14332.611892] LustreError: 444340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14333.020059] Lustre: server umount lustre-MDT0001 complete [14346.595263] Lustre: server umount lustre-OST0000 complete [14358.195827] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 04:58:02 (1788944282) [14367.387212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14368.138852] LustreError: 448347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14368.180770] LustreError: 448347:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14368.364224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14373.360877] LustreError: 448348:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14374.187541] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [14378.476477] LustreError: 448347:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14383.609750] LustreError: 448348:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14385.824347] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14386.329306] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14391.935665] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [14400.189631] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14406.391415] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14426.431769] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14426.712846] Lustre: Found index 0 for lustre-OST0000, updating log [14426.955716] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14434.695889] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [14439.235499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14443.521287] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14449.643714] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14449.647765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14449.657268] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14449.667226] Lustre: Skipped 1 previous similar message [14449.685521] Lustre: Skipped 1 previous similar message [14454.985284] Lustre: server umount lustre-OST0000 complete [14472.162498] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14472.196583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14472.210608] Lustre: Skipped 3 previous similar messages [14476.322224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14476.333040] Lustre: Skipped 1 previous similar message [14477.190639] Lustre: server umount lustre-MDT0000 complete [14477.355997] LustreError: 448343:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14477.373352] LustreError: 448343:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14481.828338] LustreError: 448327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788944408 with bad export cookie 14886028289535223764 [14481.830617] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14481.847551] LustreError: 448327:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14482.152474] Lustre: server umount lustre-MDT0001 complete [14489.445145] LNet: 1 local NIs in recovery (showing 1): 192.168.204.113@tcp [14492.333829] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 05:00:16 (1788944416) [15113.259308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15113.722575] LustreError: 454771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15113.831338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15118.819238] LustreError: 454772:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15119.018623] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [15123.942316] LustreError: 454771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15128.943791] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15129.443468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15134.037365] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [15140.739683] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15146.314299] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15166.579907] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15166.783800] Lustre: Found index 0 for lustre-OST0000, updating log [15167.013378] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15168.416442] LNet: 1 local NIs in recovery (showing 1): 192.168.204.113@tcp [15172.390407] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [15179.819665] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15182.793782] Lustre: server umount lustre-OST0000 complete [15187.205404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15187.208661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15187.239159] Lustre: Skipped 2 previous similar messages [15187.261669] Lustre: Skipped 1 previous similar message [15188.258946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15188.263913] Lustre: Skipped 1 previous similar message [15190.307186] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15190.317332] Lustre: Skipped 3 previous similar messages [15193.769436] Lustre: server umount lustre-MDT0000 complete [15194.408169] LustreError: 456016:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15194.433471] LustreError: 456016:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [15197.259964] LustreError: 454753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945124 with bad export cookie 14886028289535224849 [15197.266233] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15197.279078] LustreError: 454753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15197.544409] Lustre: server umount lustre-MDT0001 complete [15227.602546] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [15228.832638] LNet: 1 local NIs in recovery (showing 1): 192.168.204.113@tcp [15237.382268] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [15275.820409] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 05:13:19 (1788945199) [15293.971518] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [15296.941172] Key type lgssc unregistered [15297.367798] LNet: 461381:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15297.378266] LNetError: 461381:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15427.438773] LNet: Removed LNI 192.168.204.113@tcp [15428.596964] Key type .llcrypt unregistered [15428.598633] Key type ._llcrypt unregistered [15443.732796] Key type ._llcrypt registered [15443.734206] Key type .llcrypt registered [15443.859345] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [15444.614693] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15444.625429] alg: No test for adler32 (adler32-zlib) [15445.681091] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [15445.923773] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [15447.594267] Key type lgssc registered [15448.851979] Lustre: Echo OBD driver; http://www.lustre.org/ [15452.770652] LNet: Added LNI 192.168.204.113@tcp100 [8/256/0/180] [15453.604966] LNet: Added LNI 192.168.204.113@tcp101 [8/256/0/180] [15454.612716] LNet: Added LNI 192.168.204.113@tcp102 [8/256/0/180] [15455.426592] LNet: Added LNI 192.168.204.113@tcp103 [8/256/0/180] [15456.113495] LNet: Added LNI 192.168.204.113@tcp104 [8/256/0/180] [15457.098482] LNet: Added LNI 192.168.204.113@tcp105 [8/256/0/180] [15458.018111] LNet: Added LNI 192.168.204.113@tcp106 [8/256/0/180] [15459.103529] LNet: Added LNI 192.168.204.113@tcp107 [8/256/0/180] [15460.111368] LNet: Added LNI 192.168.204.113@tcp108 [8/256/0/180] [15461.191841] LNet: Added LNI 192.168.204.113@tcp109 [8/256/0/180] [15462.124489] LNet: Added LNI 192.168.204.113@tcp110 [8/256/0/180] [15462.929141] LNet: Added LNI 192.168.204.113@tcp111 [8/256/0/180] [15463.808421] LNet: Added LNI 192.168.204.113@tcp112 [8/256/0/180] [15464.514826] LNet: Added LNI 192.168.204.113@tcp113 [8/256/0/180] [15465.389991] LNet: Added LNI 192.168.204.113@tcp114 [8/256/0/180] [15466.452930] LNet: Added LNI 192.168.204.113@tcp115 [8/256/0/180] [15467.398085] LNet: Added LNI 192.168.204.113@tcp116 [8/256/0/180] [15468.332181] LNet: Added LNI 192.168.204.113@tcp117 [8/256/0/180] [15469.522042] LNet: Added LNI 192.168.204.113@tcp118 [8/256/0/180] [15470.723957] LNet: Added LNI 192.168.204.113@tcp119 [8/256/0/180] [15472.257858] LNet: Added LNI 192.168.204.113@tcp120 [8/256/0/180] [15473.608025] LNet: Added LNI 192.168.204.113@tcp121 [8/256/0/180] [15474.670960] LNet: Added LNI 192.168.204.113@tcp122 [8/256/0/180] [15475.747833] LNet: Added LNI 192.168.204.113@tcp123 [8/256/0/180] [15476.559146] LNet: Added LNI 192.168.204.113@tcp124 [8/256/0/180] [15477.356374] LNet: Added LNI 192.168.204.113@tcp125 [8/256/0/180] [15478.366809] LNet: Added LNI 192.168.204.113@tcp126 [8/256/0/180] [15479.336577] LNet: Added LNI 192.168.204.113@tcp127 [8/256/0/180] [15480.228072] LNet: Added LNI 192.168.204.113@tcp128 [8/256/0/180] [15481.124679] LNet: Added LNI 192.168.204.113@tcp129 [8/256/0/180] [15482.009728] LNet: Added LNI 192.168.204.113@tcp130 [8/256/0/180] [15482.852221] LNet: Added LNI 192.168.204.113@tcp131 [8/256/0/180] [15483.699434] LNet: Added LNI 192.168.204.113@tcp132 [8/256/0/180] [15484.589299] LNet: Added LNI 192.168.204.113@tcp133 [8/256/0/180] [15485.444760] LNet: Added LNI 192.168.204.113@tcp134 [8/256/0/180] [15486.267475] LNet: Added LNI 192.168.204.113@tcp135 [8/256/0/180] [15492.038662] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15492.063519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15493.177768] Lustre: 464411:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.113@tcp: can't add NID 192.168.204.113@tcp130: rc = -75 [15493.259708] LNetError: 464431:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [15493.277555] LNetError: Protocol error connecting to 192.168.204.113@tcp121 at host 192.168.204.113:988: is it running a compatible version of Lustre? [15493.289080] LNetError: Skipped 4 previous similar messages [15493.415318] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15493.444492] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15493.533484] Lustre: lustre-MDT0000: new disk, initializing [15493.609626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15493.625963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15494.048462] LNet: 6 local NIs in recovery (showing 6): 192.168.204.113@tcp124, 192.168.204.113@tcp125, 192.168.204.113@tcp128, 192.168.204.113@tcp127, 192.168.204.113@tcp129, 192.168.204.113@tcp121 [15494.052866] LNetError: 461869:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [15494.079094] LNetError: 461869:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [15494.095725] LNetError: Protocol error connecting to 192.168.204.113@tcp122 at host 192.168.204.113:988: is it running a compatible version of Lustre? [15494.102407] LNetError: Skipped 4 previous similar messages [15497.755968] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [15508.123028] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15508.240775] Lustre: 464427:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15508.285391] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15508.288816] Lustre: Skipped 1 previous similar message [15508.347871] Lustre: lustre-MDT0001: new disk, initializing [15508.383508] Lustre: 465229:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.113@tcp: can't add NID 192.168.204.113@tcp130: rc = -75 [15508.395210] Lustre: 465229:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15508.501055] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15508.525627] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15508.531433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15510.853502] Lustre: 465354:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.113@tcp: can't add NID 192.168.204.113@tcp130: rc = -75 [15510.868316] Lustre: 465354:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15513.390624] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [15521.058779] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15524.770335] LNetError: 464440:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [15524.781202] LNetError: 464440:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15524.790815] LNetError: Protocol error connecting to 192.168.204.113@tcp123 at host 192.168.204.113:988: is it running a compatible version of Lustre? [15526.137383] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15526.818553] LNetError: 464436:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [15526.831431] LNetError: Protocol error connecting to 192.168.204.113@tcp126 at host 192.168.204.113:988: is it running a compatible version of Lustre? [15530.914499] LNetError: 461870:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [15530.926490] LNetError: 461870:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15530.934271] LNetError: Protocol error connecting to 192.168.204.113@tcp119 at host 192.168.204.113:988: is it running a compatible version of Lustre? [15530.953519] LNetError: Skipped 2 previous similar messages [15532.036979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15532.402782] Lustre: lustre-OST0000: new disk, initializing [15532.409373] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15532.414163] Lustre: 466401:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15532.493253] Lustre: 466406:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.113@tcp: can't add NID 192.168.204.113@tcp130: rc = -75 [15532.497788] Lustre: 466406:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15532.549447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15537.689095] Lustre: 466696:0:(lustre_peer.c:124:class_add_uuid()) 192.168.204.113@tcp: can't add NID 192.168.204.113@tcp130: rc = -75 [15537.694105] Lustre: 466696:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15537.731447] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15537.740281] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15537.842206] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15538.418348] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [15541.155303] LNetError: 461869:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [15541.174179] LNetError: 461869:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 3 previous similar messages [15541.182302] LNetError: Protocol error connecting to 192.168.204.113@tcp124 at host 192.168.204.113:988: is it running a compatible version of Lustre? [15541.187475] LNetError: Skipped 3 previous similar messages [15546.637175] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15553.528841] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15553.540065] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15553.541709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15553.556285] Lustre: Skipped 1 previous similar message [15553.574816] Lustre: Skipped 1 previous similar message [15557.537525] LNetError: 461869:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [15557.560936] LNetError: 461869:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [15557.575472] LNetError: Protocol error connecting to 192.168.204.113@tcp122 at host 192.168.204.113:988: is it running a compatible version of Lustre? [15557.587079] LNetError: Skipped 8 previous similar messages [15558.626969] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15558.637805] Lustre: Skipped 1 previous similar message [15563.751336] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15563.761196] Lustre: Skipped 1 previous similar message [15567.328135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15567.554859] Lustre: server umount lustre-OST0000 complete [15573.987372] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15573.991328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15574.004847] Lustre: Skipped 1 previous similar message [15574.016703] Lustre: Skipped 1 previous similar message [15579.105752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15579.118693] Lustre: Skipped 1 previous similar message [15586.274576] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15586.461583] Lustre: server umount lustre-MDT0000 complete [15590.308489] LNetError: 464431:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.204.113 [15590.318222] LNetError: 464431:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15590.322483] LNetError: Protocol error connecting to 192.168.204.113@tcp126 at host 192.168.204.113:988: is it running a compatible version of Lustre? [15590.334458] LNetError: Skipped 7 previous similar messages [15590.663858] LustreError: 465228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788945517 with bad export cookie 4732766532736490068 [15590.673735] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15590.673991] LustreError: 465228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15591.005292] Lustre: server umount lustre-MDT0001 complete [15601.562788] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [15604.184105] Key type lgssc unregistered [15604.583241] LNet: 468317:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15604.592676] LNetError: 468317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15604.602506] LNet: Removed LNI 192.168.204.113@tcp [15605.664186] LNet: Removed LNI 192.168.204.113@tcp100 [15605.670176] LNetError: 468317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15605.681261] LNetError: 468317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15605.691690] LNet: Removed LNI 192.168.204.113@tcp101 [15606.752203] LNet: Removed LNI 192.168.204.113@tcp102 [15606.773803] LNetError: 468317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15606.781576] LNetError: 468317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15606.794052] LNet: Removed LNI 192.168.204.113@tcp103 [15607.842019] LNet: Removed LNI 192.168.204.113@tcp104 [15608.929044] LNet: Removed LNI 192.168.204.113@tcp105 [15608.932958] LNetError: 468317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15608.938719] LNetError: 468317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15608.944738] LNet: Removed LNI 192.168.204.113@tcp106 [15609.953898] LNet: Removed LNI 192.168.204.113@tcp107 [15610.976188] LNet: Removed LNI 192.168.204.113@tcp108 [15612.001413] LNet: Removed LNI 192.168.204.113@tcp109 [15613.025552] LNet: Removed LNI 192.168.204.113@tcp110 [15613.027565] LNetError: 468317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15613.036099] LNetError: 468317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15613.049776] LNet: Removed LNI 192.168.204.113@tcp111 [15614.112169] LNet: Removed LNI 192.168.204.113@tcp112 [15615.136370] LNet: Removed LNI 192.168.204.113@tcp113 [15616.161378] LNet: Removed LNI 192.168.204.113@tcp114 [15617.184434] LNet: Removed LNI 192.168.204.113@tcp115 [15618.211479] LNet: Removed LNI 192.168.204.113@tcp116 [15619.232485] LNet: Removed LNI 192.168.204.113@tcp117 [15620.256164] LNet: Removed LNI 192.168.204.113@tcp118 [15621.280169] LNet: Removed LNI 192.168.204.113@tcp119 [15621.283365] LNetError: 468317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15621.289675] LNetError: 468317:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15621.297846] LNet: Removed LNI 192.168.204.113@tcp120 [15622.304127] LNet: Removed LNI 192.168.204.113@tcp121 [15623.328549] LNet: Removed LNI 192.168.204.113@tcp122 [15624.352316] LNet: Removed LNI 192.168.204.113@tcp123 [15625.376592] LNet: Removed LNI 192.168.204.113@tcp124 [15626.400154] LNet: Removed LNI 192.168.204.113@tcp125 [15627.424184] LNet: Removed LNI 192.168.204.113@tcp126 [15628.448265] LNet: Removed LNI 192.168.204.113@tcp127 [15629.472244] LNet: Removed LNI 192.168.204.113@tcp128 [15630.496579] LNet: Removed LNI 192.168.204.113@tcp129 [15630.505275] LNet: Removed LNI 192.168.204.113@tcp130 [15630.509769] LNet: Removed LNI 192.168.204.113@tcp131 [15630.515039] LNet: Removed LNI 192.168.204.113@tcp132 [15630.519746] LNet: Removed LNI 192.168.204.113@tcp133 [15630.524693] LNet: Removed LNI 192.168.204.113@tcp134 [15630.537089] LNet: Removed LNI 192.168.204.113@tcp135 [15631.335068] Key type .llcrypt unregistered [15631.337115] Key type ._llcrypt unregistered [15654.005124] Key type ._llcrypt registered [15654.007209] Key type .llcrypt registered [15654.090185] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [15667.970555] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [15669.339265] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15669.365040] alg: No test for adler32 (adler32-zlib) [15670.499456] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [15670.725712] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [15672.424236] Key type lgssc registered [15673.391161] Lustre: Echo OBD driver; http://www.lustre.org/ [15708.767198] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 05:20:33 (1788945633) [15725.828312] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [15728.006291] Key type lgssc unregistered [15728.334361] LNet: 472398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15728.361719] LNetError: 472398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15728.384691] LNet: Removed LNI 192.168.204.113@tcp [15729.152868] Key type .llcrypt unregistered [15729.155216] Key type ._llcrypt unregistered [15743.487864] Key type ._llcrypt registered [15743.490974] Key type .llcrypt registered [15743.593697] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [15744.434535] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15744.466051] alg: No test for adler32 (adler32-zlib) [15745.455769] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [15745.654034] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [15747.320825] Key type lgssc registered [15748.286300] Lustre: Echo OBD driver; http://www.lustre.org/ [15756.642783] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15756.671334] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15757.834564] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15757.863250] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15757.918697] Lustre: lustre-MDT0000: new disk, initializing [15758.025941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15758.037766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15761.345911] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [15769.381828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15769.469195] Lustre: 473605:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15769.513875] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15769.517686] Lustre: Skipped 1 previous similar message [15769.595670] Lustre: lustre-MDT0001: new disk, initializing [15769.701027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15769.746449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15769.756086] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15772.793113] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [15780.466840] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15785.074703] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15791.076240] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15791.264747] Lustre: lustre-OST0000: new disk, initializing [15791.267534] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15791.270846] Lustre: 475576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15791.334946] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15796.691706] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [15797.812637] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15797.818890] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15797.895828] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15804.559676] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15809.202487] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing lnet_if_list [15815.637173] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15815.681970] LNet: Added LNI 192.168.204.113@tcp10 [8/256/0/180] [15822.284925] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15822.330340] LNet: Added LNI 192.168.204.113@tcp11 [8/256/0/180] [15827.426806] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15827.454644] LNet: Added LNI 192.168.204.113@tcp12 [8/256/0/180] [15832.318815] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15832.378285] LNet: Added LNI 192.168.204.113@tcp13 [8/256/0/180] [15837.273613] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15837.328357] LNet: Added LNI 192.168.204.113@tcp14 [8/256/0/180] [15841.541615] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15841.574974] LNet: Added LNI 192.168.204.113@tcp15 [8/256/0/180] [15846.015893] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15846.050749] LNet: Added LNI 192.168.204.113@tcp16 [8/256/0/180] [15850.497148] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15850.530636] LNet: Added LNI 192.168.204.113@tcp17 [8/256/0/180] [15854.707194] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15854.739621] LNet: Added LNI 192.168.204.113@tcp18 [8/256/0/180] [15858.316845] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15858.347079] LNet: Added LNI 192.168.204.113@tcp19 [8/256/0/180] [15862.991763] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15863.019750] LNet: Added LNI 192.168.204.113@tcp20 [8/256/0/180] [15867.208874] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15867.253369] LNet: Added LNI 192.168.204.113@tcp21 [8/256/0/180] [15873.061480] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15873.119492] LNet: Added LNI 192.168.204.113@tcp22 [8/256/0/180] [15880.606084] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15880.634490] LNet: Added LNI 192.168.204.113@tcp23 [8/256/0/180] [15886.640194] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15886.697376] LNet: Added LNI 192.168.204.113@tcp24 [8/256/0/180] [15892.297713] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15892.318627] LNet: Added LNI 192.168.204.113@tcp25 [8/256/0/180] [15897.390075] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15897.425379] LNet: Added LNI 192.168.204.113@tcp26 [8/256/0/180] [15902.708228] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15902.748221] LNet: Added LNI 192.168.204.113@tcp27 [8/256/0/180] [15908.698513] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15908.745413] LNet: Added LNI 192.168.204.113@tcp28 [8/256/0/180] [15914.321586] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15914.352487] LNet: Added LNI 192.168.204.113@tcp29 [8/256/0/180] [15920.703885] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15920.734742] LNet: Added LNI 192.168.204.113@tcp30 [8/256/0/180] [15927.390859] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15927.423149] LNet: Added LNI 192.168.204.113@tcp31 [8/256/0/180] [15932.987960] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15933.024280] LNet: Added LNI 192.168.204.113@tcp32 [8/256/0/180] [15939.297503] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15939.372320] LNet: Added LNI 192.168.204.113@tcp33 [8/256/0/180] [15946.516185] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15946.544107] LNet: Added LNI 192.168.204.113@tcp34 [8/256/0/180] [15954.509414] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15954.548842] LNet: Added LNI 192.168.204.113@tcp35 [8/256/0/180] [15961.245044] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15961.262711] LNet: Added LNI 192.168.204.113@tcp36 [8/256/0/180] [15968.308727] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15968.348856] LNet: Added LNI 192.168.204.113@tcp37 [8/256/0/180] [15975.621793] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15975.683764] LNet: Added LNI 192.168.204.113@tcp38 [8/256/0/180] [15981.711814] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15981.734726] LNet: Added LNI 192.168.204.113@tcp39 [8/256/0/180] [15989.281774] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15989.305154] LNet: Added LNI 192.168.204.113@tcp40 [8/256/0/180] [15997.873391] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15997.901081] LNet: Added LNI 192.168.204.113@tcp41 [8/256/0/180] [16007.691299] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16007.731598] LNet: Added LNI 192.168.204.113@tcp42 [8/256/0/180] [16015.416949] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16015.468211] LNet: Added LNI 192.168.204.113@tcp43 [8/256/0/180] [16022.081911] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16022.104425] LNet: Added LNI 192.168.204.113@tcp44 [8/256/0/180] [16028.696392] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16028.722042] LNet: Added LNI 192.168.204.113@tcp45 [8/256/0/180] [16035.332252] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16035.359830] LNet: Added LNI 192.168.204.113@tcp46 [8/256/0/180] [16042.297804] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16042.330962] LNet: Added LNI 192.168.204.113@tcp47 [8/256/0/180] [16049.549031] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16049.583848] LNet: Added LNI 192.168.204.113@tcp48 [8/256/0/180] [16058.625303] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16058.669514] LNet: Added LNI 192.168.204.113@tcp49 [8/256/0/180] [16068.265501] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16068.326971] LNet: Added LNI 192.168.204.113@tcp50 [8/256/0/180] [16087.009620] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16087.017734] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16087.047721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16090.082917] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16090.091272] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16090.122831] Lustre: Skipped 1 previous similar message [16092.769203] Lustre: server umount lustre-MDT0000 complete [16098.102271] LustreError: 473598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946024 with bad export cookie 559536990127099952 [16098.111786] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16098.565776] Lustre: server umount lustre-MDT0001 complete [16116.707643] Lustre: 472878:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788946027/real 1788946027] req@000000008e0d5187 x1875845497126144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788946043 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16116.728622] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16119.456212] Lustre: 472879:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788946030/real 1788946030] req@00000000cba4160a x1875845497126272/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788946046 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16120.894786] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [16121.825067] Lustre: 472881:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788946032/real 1788946032] req@000000005b3d5c6b x1875845497126400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788946048 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16123.872105] Lustre: 472881:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788946034/real 1788946034] req@0000000085f6561d x1875845497126656/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788946050 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16128.992402] Lustre: 472878:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788946039/real 1788946039] req@00000000f2f1e49d x1875845497126784/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788946055 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16131.235869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16142.700093] LustreError: 475569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16142.713687] LustreError: 475569:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [16142.759555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16145.828161] LustreError: 475570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16145.855958] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16146.850204] LustreError: 475569:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16147.759536] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16152.038251] LustreError: 475570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16157.159135] LustreError: 475570:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16157.226790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16157.701556] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16161.780122] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16162.188586] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16170.154770] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16175.225845] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16192.487308] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16192.492037] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16192.506430] Lustre: Skipped 1 previous similar message [16192.511448] Lustre: Skipped 2 previous similar messages [16197.609837] LustreError: 501457:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16197.632298] LustreError: 501457:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16197.722152] Lustre: server umount lustre-OST0000 complete [16202.722519] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16202.728410] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16202.737845] Lustre: Skipped 1 previous similar message [16207.259661] Lustre: server umount lustre-MDT0000 complete [16210.943113] LustreError: 474401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946137 with bad export cookie 559536990127102458 [16210.946342] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16210.960988] LustreError: 474401:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [16211.415784] Lustre: server umount lustre-MDT0001 complete [16221.061461] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [16223.702485] Key type lgssc unregistered [16224.057529] LNet: 502601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16224.063471] LNetError: 502601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16224.068675] LNet: Removed LNI 192.168.204.113@tcp [16224.071695] LNet: Removed LNI 192.168.204.113@tcp10 [16224.075267] LNet: Removed LNI 192.168.204.113@tcp11 [16224.078427] LNet: Removed LNI 192.168.204.113@tcp12 [16224.081773] LNet: Removed LNI 192.168.204.113@tcp13 [16224.083344] LNet: Removed LNI 192.168.204.113@tcp14 [16224.085368] LNet: Removed LNI 192.168.204.113@tcp15 [16224.090298] LNet: Removed LNI 192.168.204.113@tcp16 [16224.093360] LNet: Removed LNI 192.168.204.113@tcp17 [16224.098442] LNet: Removed LNI 192.168.204.113@tcp18 [16224.101570] LNet: Removed LNI 192.168.204.113@tcp19 [16224.103800] LNet: Removed LNI 192.168.204.113@tcp20 [16224.106742] LNet: Removed LNI 192.168.204.113@tcp21 [16224.109625] LNet: Removed LNI 192.168.204.113@tcp22 [16224.112361] LNet: Removed LNI 192.168.204.113@tcp23 [16224.115529] LNet: Removed LNI 192.168.204.113@tcp24 [16224.118406] LNet: Removed LNI 192.168.204.113@tcp25 [16224.122191] LNet: Removed LNI 192.168.204.113@tcp26 [16224.124650] LNet: Removed LNI 192.168.204.113@tcp27 [16224.128493] LNet: Removed LNI 192.168.204.113@tcp28 [16224.130971] LNet: Removed LNI 192.168.204.113@tcp29 [16224.132926] LNet: Removed LNI 192.168.204.113@tcp30 [16224.135428] LNet: Removed LNI 192.168.204.113@tcp31 [16224.138598] LNet: Removed LNI 192.168.204.113@tcp32 [16224.143930] LNet: Removed LNI 192.168.204.113@tcp33 [16224.147984] LNet: Removed LNI 192.168.204.113@tcp34 [16224.152314] LNet: Removed LNI 192.168.204.113@tcp35 [16224.158088] LNet: Removed LNI 192.168.204.113@tcp36 [16224.160095] LNet: Removed LNI 192.168.204.113@tcp37 [16224.162948] LNet: Removed LNI 192.168.204.113@tcp38 [16224.168606] LNet: Removed LNI 192.168.204.113@tcp39 [16224.179684] LNet: Removed LNI 192.168.204.113@tcp40 [16224.195967] LNet: Removed LNI 192.168.204.113@tcp41 [16224.199044] LNet: Removed LNI 192.168.204.113@tcp42 [16224.204109] LNet: Removed LNI 192.168.204.113@tcp43 [16224.207487] LNet: Removed LNI 192.168.204.113@tcp44 [16224.211689] LNet: Removed LNI 192.168.204.113@tcp45 [16224.214694] LNet: Removed LNI 192.168.204.113@tcp46 [16224.217961] LNet: Removed LNI 192.168.204.113@tcp47 [16224.225671] LNet: Removed LNI 192.168.204.113@tcp48 [16224.239477] LNet: Removed LNI 192.168.204.113@tcp49 [16224.259211] LNet: Removed LNI 192.168.204.113@tcp50 [16225.021939] Key type .llcrypt unregistered [16225.024671] Key type ._llcrypt unregistered [16244.739700] Key type ._llcrypt registered [16244.742691] Key type .llcrypt registered [16244.844316] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [16259.330695] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [16260.225456] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16260.458946] alg: No test for adler32 (adler32-zlib) [16261.498191] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [16261.696662] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [16263.352358] Key type lgssc registered [16264.352562] Lustre: Echo OBD driver; http://www.lustre.org/ [16303.644463] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16303.697631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16304.973635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16305.037376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16305.174185] Lustre: lustre-MDT0000: new disk, initializing [16305.315861] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16305.347987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16309.572485] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16320.038766] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16320.190468] Lustre: 505796:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16320.266613] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16320.272279] Lustre: Skipped 1 previous similar message [16320.279285] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [16320.418255] Lustre: lustre-MDT0001: new disk, initializing [16320.531553] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16320.578894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16320.590278] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16324.682101] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16334.257417] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16340.265291] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16348.420981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16348.564935] Lustre: *** cfs_fail_loc=90f, val=0*** [16348.663629] Lustre: lustre-OST0000: new disk, initializing [16348.669286] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16348.683698] Lustre: 507811:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16348.775663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16354.394501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16354.418276] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16354.528787] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16356.504881] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16368.968517] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16380.385386] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16380.390517] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16380.399532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16380.400645] Lustre: Skipped 1 previous similar message [16380.415362] Lustre: Skipped 1 previous similar message [16383.879424] Lustre: server umount lustre-OST0000 complete [16390.637236] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16390.639853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16390.643777] Lustre: Skipped 1 previous similar message [16390.652462] Lustre: Skipped 1 previous similar message [16395.747968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16395.755784] Lustre: Skipped 1 previous similar message [16400.868303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16400.875928] Lustre: Skipped 1 previous similar message [16402.400554] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16402.625991] Lustre: server umount lustre-MDT0000 complete [16406.401506] LustreError: 506605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946333 with bad export cookie 3312583955588801731 [16406.414090] LustreError: 506605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16406.421942] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16406.938625] Lustre: server umount lustre-MDT0001 complete [16421.467300] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [16424.973474] Key type lgssc unregistered [16425.332138] LNet: 509718:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16425.345832] LNetError: 509718:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16425.371332] LNet: Removed LNI 192.168.204.113@tcp [16426.652433] Key type .llcrypt unregistered [16426.671165] Key type ._llcrypt unregistered [16447.587996] Key type ._llcrypt registered [16447.592919] Key type .llcrypt registered [16447.734724] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [16448.295090] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16448.341388] alg: No test for adler32 (adler32-zlib) [16449.324619] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [16449.543636] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [16451.208662] Key type lgssc registered [16452.605779] Lustre: Echo OBD driver; http://www.lustre.org/ [16459.939747] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16459.968585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16461.698351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16467.947045] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16472.040482] LustreError: 510988:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16472.061805] LustreError: 510988:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16477.166171] LustreError: 510987:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16478.625544] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16478.965778] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16483.943891] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16492.657937] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16498.975309] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16506.733801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16506.863106] LustreError: 512810:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16507.152311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16513.624976] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16522.103851] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16532.615249] loop0: detected capacity change from 0 to 204800000 [16532.826446] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16532.845321] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16532.855821] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16539.911404] loop0: detected capacity change from 0 to 204800000 [16540.175866] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16540.199936] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16540.214840] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16546.856181] loop0: detected capacity change from 0 to 204800000 [16553.141825] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16553.305915] Lustre: fs15246-OST0000: new disk, initializing [16553.310548] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16553.316733] Lustre: 514582:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16553.396121] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16560.281921] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16570.572389] loop1: detected capacity change from 0 to 204800000 [16577.459912] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16577.626943] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16577.730802] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16577.893600] Lustre: fs15246-MDT0000: new disk, initializing [16578.074063] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16578.129936] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16578.322586] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16579.365745] Lustre: 515704:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000af21b5dd x1875846235212416/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788946522 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16583.460339] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16588.560994] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16588.588324] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16588.610377] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16605.680349] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16605.709644] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16610.796455] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16611.280695] Lustre: server umount fs15246-MDT0000 complete [16615.910364] LustreError: 510984:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16621.034054] LustreError: 510984:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16622.137287] Lustre: server umount fs15246-OST0000 complete [16635.375929] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16635.388075] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16635.411029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16636.393411] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16641.103179] Lustre: server umount lustre-OST0000 complete [16646.626977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16646.640237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16646.642035] Lustre: Skipped 1 previous similar message [16646.673198] Lustre: Skipped 3 previous similar messages [16651.747763] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16651.765675] Lustre: Skipped 1 previous similar message [16652.653920] Lustre: server umount lustre-MDT0000 complete [16656.880621] LustreError: 513479:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16656.919402] LustreError: 513479:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16657.290628] LustreError: 510967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946584 with bad export cookie 5339910589541187058 [16657.299267] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16657.614424] Lustre: server umount lustre-MDT0001 complete [16670.277858] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [16673.348893] Key type lgssc unregistered [16673.608963] LNet: 518800:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16673.614789] LNetError: 518800:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16674.665047] LNet: Removed LNI 192.168.204.113@tcp [16675.404331] Key type .llcrypt unregistered [16675.406093] Key type ._llcrypt unregistered [16699.117173] Key type ._llcrypt registered [16699.118608] Key type .llcrypt registered [16699.259059] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [16700.210049] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16700.235041] alg: No test for adler32 (adler32-zlib) [16701.327681] Lustre: Lustre: Build Version: 2.17.58_39_gf1a8369 [16701.547333] LNet: Added LNI 192.168.204.113@tcp [8/256/0/180] [16703.176340] Key type lgssc registered [16704.712845] Lustre: Echo OBD driver; http://www.lustre.org/ [16751.240417] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [16760.228047] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [16801.131721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16801.145542] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16802.605510] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16802.649901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16802.851705] Lustre: lustre-MDT0000: new disk, initializing [16803.024856] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16803.078057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16808.292461] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16820.610689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16820.781944] Lustre: 523271:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16820.885607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16820.893783] Lustre: Skipped 1 previous similar message [16821.062652] Lustre: lustre-MDT0001: new disk, initializing [16821.290928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16821.374541] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16821.392390] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16827.234783] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16839.764572] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16846.900713] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16854.355712] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16854.635387] Lustre: lustre-OST0000: new disk, initializing [16854.643957] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16854.657893] Lustre: 525239:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16854.758887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16860.734725] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_default_debug -1 all [16862.783481] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16862.793578] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16862.896512] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16870.418523] Lustre: DEBUG MARKER: oleg413-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16874.988649] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16875.254601] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16880.154169] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16880.360818] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16883.689076] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16883.691482] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16883.713144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16883.723520] LustreError: Skipped 1 previous similar message [16883.731454] Lustre: Skipped 1 previous similar message [16888.532653] Lustre: server umount lustre-OST0000 complete [16893.921378] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16893.935943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16893.936954] Lustre: Skipped 2 previous similar messages [16893.958567] Lustre: Skipped 1 previous similar message [16899.045527] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16899.062830] Lustre: Skipped 1 previous similar message [16899.721277] Lustre: server umount lustre-MDT0000 complete [16903.528119] LustreError: 524066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788946830 with bad export cookie 1720493574733703910 [16903.530306] LustreError: MGC192.168.204.113@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16903.541111] LustreError: 524066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16903.878636] Lustre: server umount lustre-MDT0001 complete [16932.088040] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing set_hostid [16944.862275] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing load_modules_local [16981.574681] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16702 sec ========= 05:41:45 (1788946905) [16983.469317] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 05:41:47 (1788946907) === [16987.459355] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 05:41:51 (1788946911) === [17030.246458] Lustre: DEBUG MARKER: oleg413-server.virtnet: executing unload_modules_local [17032.741674] Key type lgssc unregistered [17033.007664] LNet: 531882:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17033.014330] LNetError: 531882:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17034.087108] LNet: Removed LNI 192.168.204.113@tcp [17034.797120] Key type .llcrypt unregistered [17034.798765] Key type ._llcrypt unregistered