[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 460621409 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002403] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004014] kvm-guest: setup PV IPIs [ 0.007000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008010] pid_max: default: 32768 minimum: 301 [ 0.009133] LSM: Security Framework initializing [ 0.010071] Yama: becoming mindful. [ 0.011035] SELinux: Initializing. [ 0.012069] *** VALIDATE selinux *** [ 0.020818] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.024821] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.025149] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026110] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027112] *** VALIDATE tmpfs *** [ 0.028428] *** VALIDATE proc *** [ 0.029263] *** VALIDATE cgroup *** [ 0.030010] *** VALIDATE cgroup2 *** [ 0.031207] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.032158] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.033009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.034029] Spectre V2 : User space: Vulnerable [ 0.035007] Speculative Store Bypass: Vulnerable [ 0.038268] debug: unmapping init [mem 0xffffffffb6e59000-0xffffffffb6e60fff] [ 0.040176] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.041727] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.042024] ... version: 2 [ 0.043013] ... bit width: 48 [ 0.044012] ... generic registers: 4 [ 0.045014] ... value mask: 0000ffffffffffff [ 0.046018] ... max period: 00007fffffffffff [ 0.047016] ... fixed-purpose events: 3 [ 0.048013] ... event mask: 000000070000000f [ 0.050229] rcu: Hierarchical SRCU implementation. [ 0.052438] smp: Bringing up secondary CPUs ... [ 0.053497] x86: Booting SMP configuration: [ 0.054027] .... node #0, CPUs: #1 #2 #3 [ 0.057686] smp: Brought up 1 node, 4 CPUs [ 0.059015] smpboot: Max logical packages: 1 [ 0.060022] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.228580] node 0 deferred pages initialised in 165ms [ 0.231149] devtmpfs: initialized [ 0.232217] x86/mm: Memory block size: 128MB [ 0.234663] gcov: version magic: 0x41383552 [ 0.235606] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.239187] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.241457] pinctrl core: initialized pinctrl subsystem [ 0.244272] [ 0.244947] ************************************************************* [ 0.248020] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.251017] ** ** [ 0.253014] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.255018] ** ** [ 0.258014] ** This means that this kernel is built to expose internal ** [ 0.261018] ** IOMMU data structures, which may compromise security on ** [ 0.263017] ** your system. ** [ 0.266019] ** ** [ 0.270019] ** If you see this message and you are not debugging the ** [ 0.273015] ** kernel, report this immediately to your vendor! ** [ 0.275016] ** ** [ 0.278016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.280017] ************************************************************* [ 0.283824] NET: Registered protocol family 16 [ 0.286509] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.289084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.292089] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.297021] cpuidle: using governor menu [ 0.298784] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.301706] PCI: Using configuration type 1 for base access [ 0.304156] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.313176] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.316024] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.320065] cryptd: max_cpu_qlen set to 1000 [ 0.323309] ACPI: Added _OSI(Module Device) [ 0.325022] ACPI: Added _OSI(Processor Device) [ 0.327016] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.329020] ACPI: Added _OSI(Processor Aggregator Device) [ 0.334371] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.341452] ACPI: Interpreter enabled [ 0.343054] ACPI: PM: (supports S0 S3 S4 S5) [ 0.345014] ACPI: Using IOAPIC for interrupt routing [ 0.347101] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.351491] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.363323] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.365050] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.368031] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.372143] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.377496] acpiphp: Slot [2] registered [ 0.379210] acpiphp: Slot [5] registered [ 0.381214] acpiphp: Slot [6] registered [ 0.382164] acpiphp: Slot [7] registered [ 0.384169] acpiphp: Slot [8] registered [ 0.387205] acpiphp: Slot [9] registered [ 0.389187] acpiphp: Slot [10] registered [ 0.391189] acpiphp: Slot [3] registered [ 0.393127] acpiphp: Slot [4] registered [ 0.394127] acpiphp: Slot [11] registered [ 0.396154] acpiphp: Slot [12] registered [ 0.398177] acpiphp: Slot [13] registered [ 0.400200] acpiphp: Slot [14] registered [ 0.402191] acpiphp: Slot [15] registered [ 0.404223] acpiphp: Slot [16] registered [ 0.406182] acpiphp: Slot [17] registered [ 0.407141] acpiphp: Slot [18] registered [ 0.409151] acpiphp: Slot [19] registered [ 0.411145] acpiphp: Slot [20] registered [ 0.412136] acpiphp: Slot [21] registered [ 0.414128] acpiphp: Slot [22] registered [ 0.416152] acpiphp: Slot [23] registered [ 0.417224] acpiphp: Slot [24] registered [ 0.419138] acpiphp: Slot [25] registered [ 0.421139] acpiphp: Slot [26] registered [ 0.422203] acpiphp: Slot [27] registered [ 0.424130] acpiphp: Slot [28] registered [ 0.426178] acpiphp: Slot [29] registered [ 0.427165] acpiphp: Slot [30] registered [ 0.429110] acpiphp: Slot [31] registered [ 0.430114] PCI host bridge to bus 0000:00 [ 0.431029] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.434046] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.437045] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.440034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.443034] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.447038] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.449210] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.452232] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.456536] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.467019] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.472066] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.475026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.477027] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.481024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.484486] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.486886] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.491066] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.494925] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.500024] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.515035] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.521025] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.527480] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.537021] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.544018] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.569023] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.585534] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.593022] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.601021] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.621026] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.632992] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.644022] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.654021] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.671033] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.681828] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.693022] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.704023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.730022] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.741239] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.748020] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.753018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.768018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.779145] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.785019] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.796022] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.818024] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.831127] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.834447] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.836387] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.839490] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.842297] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.847114] iommu: Default domain type: Passthrough [ 0.849583] SCSI subsystem initialized [ 0.851270] ACPI: bus type USB registered [ 0.853144] usbcore: registered new interface driver usbfs [ 0.855102] usbcore: registered new interface driver hub [ 0.857102] usbcore: registered new device driver usb [ 0.859253] pps_core: LinuxPPS API ver. 1 registered [ 0.861015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.864085] PTP clock support registered [ 0.867092] EDAC MC: Ver: 3.0.0 [ 0.870181] PCI: Using ACPI for IRQ routing [ 0.872025] NetLabel: Initializing [ 0.873014] NetLabel: domain hash size = 128 [ 0.874017] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.875127] NetLabel: unlabeled traffic allowed by default [ 0.878109] vgaarb: loaded [ 0.880260] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.882018] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.889414] clocksource: Switched to clocksource kvm-clock [ 1.001087] VFS: Disk quotas dquot_6.6.0 [ 1.002675] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.005370] *** VALIDATE ramfs *** [ 1.006494] *** VALIDATE hugetlbfs *** [ 1.008348] pnp: PnP ACPI init [ 1.010950] pnp: PnP ACPI: found 6 devices [ 1.029065] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.032633] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.035129] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.037715] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.040683] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.043698] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.047186] NET: Registered protocol family 2 [ 1.049899] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.055665] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.061040] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.066501] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.070662] TCP: Hash tables configured (established 65536 bind 65536) [ 1.074187] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.079294] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.083839] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.088689] NET: Registered protocol family 1 [ 1.092788] RPC: Registered named UNIX socket transport module. [ 1.094752] RPC: Registered udp transport module. [ 1.096349] RPC: Registered tcp transport module. [ 1.098160] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.100760] NET: Registered protocol family 44 [ 1.102766] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.104680] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.106973] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.109638] PCI: CLS 0 bytes, default 64 [ 1.110958] Unpacking initramfs... [ 2.560909] debug: unmapping init [mem 0xffff8be17cc54000-0xffff8be17ffbffff] [ 2.564774] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.567322] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.569498] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.058507] Initialise system trusted keyrings [ 3.060227] Key type blacklist registered [ 3.061742] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.070126] zbud: loaded [ 3.074063] *** VALIDATE nfs *** [ 3.075447] *** VALIDATE nfs4 *** [ 3.077729] pstore: using deflate compression [ 3.081555] Platform Keyring initialized [ 3.189208] NET: Registered protocol family 38 [ 3.191311] Key type asymmetric registered [ 3.193154] Asymmetric key parser 'x509' registered [ 3.195327] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.198243] io scheduler mq-deadline registered [ 3.200834] io scheduler kyber registered [ 3.202796] io scheduler bfq registered [ 3.204740] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.208473] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.210542] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.213158] ACPI: Power Button [PWRF] [ 3.218200] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.226096] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.239794] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.247494] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.268869] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.296396] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.327393] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.333268] Non-volatile memory driver v1.3 [ 3.335421] Linux agpgart interface v0.103 [ 3.367775] virtio_blk virtio1: [vda] 149768 512-byte logical blocks (76.7 MB/73.1 MiB) [ 3.372750] vda: detected capacity change from 0 to 76681216 [ 3.392323] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.396190] vdb: detected capacity change from 0 to 1073741824 [ 3.411746] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.415659] vdc: detected capacity change from 0 to 2621440000 [ 3.430122] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.432698] vdd: detected capacity change from 0 to 2621440000 [ 3.449631] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.453067] vde: detected capacity change from 0 to 4294967296 [ 3.469880] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.474474] vdf: detected capacity change from 0 to 4294967296 [ 3.484756] libphy: Fixed MDIO Bus: probed [ 3.490157] usbcore: registered new interface driver usbserial_generic [ 3.493364] usbserial: USB Serial support registered for generic [ 3.495933] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.499883] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.502099] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.505553] mousedev: PS/2 mouse device common for all mice [ 3.508910] rtc_cmos 00:05: RTC can wake from S4 [ 3.512410] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.514554] rtc_cmos 00:05: registered as rtc0 [ 3.518296] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.518635] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.521443] intel_pstate: CPU model not supported [ 3.527465] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.531192] hid: raw HID events driver (C) Jiri Kosina [ 3.533243] usbcore: registered new interface driver usbhid [ 3.535430] usbhid: USB HID core driver [ 3.537151] drop_monitor: Initializing network drop monitor service [ 3.539707] Initializing XFRM netlink socket [ 3.541738] NET: Registered protocol family 10 [ 3.545108] Segment Routing with IPv6 [ 3.546864] NET: Registered protocol family 17 [ 3.549942] mpls_gso: MPLS GSO support [ 3.556417] RAS: Correctable Errors collector initialized. [ 3.558410] AVX version of gcm_enc/dec engaged. [ 3.559788] AES CTR mode by8 optimization enabled [ 3.637098] sched_clock: Marking stable (3637010356, 0)->(4570631299, -933620943) [ 3.639556] registered taskstats version 1 [ 3.641239] Loading compiled-in X.509 certificates [ 3.643483] zswap: loaded using pool lzo/zbud [ 3.666888] Key type big_key registered [ 3.679801] Key type encrypted registered [ 3.681818] ima: No TPM chip found, activating TPM-bypass! [ 3.683938] ima: Allocated hash algorithm: sha1 [ 3.686086] ima: No architecture policies found [ 3.687621] evm: Initialising EVM extended attributes: [ 3.689462] evm: security.selinux [ 3.690983] evm: security.ima [ 3.692254] evm: security.capability [ 3.693878] evm: HMAC attrs: 0x1 [ 3.696575] rtc_cmos 00:05: setting system clock to 2026-09-05 20:57:08 UTC (1788641828) [ 3.703289] debug: unmapping init [mem 0xffffffffb7e03000-0xffffffffb7ffffff] [ 3.706595] debug: unmapping init [mem 0xffffffffb6b82000-0xffffffffb6e58fff] [ 3.717222] Write protecting the kernel read-only data: 28672k [ 3.720275] debug: unmapping init [mem 0xffffffffb5203000-0xffffffffb53fffff] [ 3.723127] debug: unmapping init [mem 0xffffffffb5b14000-0xffffffffb5bfffff] [ 3.768798] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.777445] systemd[1]: Detected virtualization kvm. [ 3.779088] systemd[1]: Detected architecture x86-64. [ 3.781621] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.809360] systemd[1]: No hostname configured. [ 3.811263] systemd[1]: Set hostname to . [ 3.813744] random: systemd: uninitialized urandom read (16 bytes read) [ 3.816398] systemd[1]: Initializing machine ID from random generator. [ 3.865030] random: ln: uninitialized urandom read (6 bytes read) [ 3.950172] random: systemd: uninitialized urandom read (16 bytes read) [ 3.952704] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.956802] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 3.966453] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Sockets. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Timers. [ OK ] Reached target Local Encrypted Volumes. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Started Memstrack Anylazing Service. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.614198] device-mapper: uevent: version 1.0.3 [ 4.616773] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook.[ 4.838063] random: fast init done Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.344147] virtio_net virtio0 ens2: renamed from eth0 [ 5.442292] scsi host0: ata_piix [ 5.494608] scsi host1: ata_piix [ 5.496710] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.500324] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.011332] dracut-initqueue[587]: RTNETLINK answers: File exists [ 10.198263] random: crng init done [ 10.199956] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.570126] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.634933] printk: systemd: 24 output lines suppressed due to ratelimiting [ 11.872364] SELinux: Disabled at runtime. [ 11.934491] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.943467] systemd[1]: Detected virtualization kvm. [ 11.945288] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.406568] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.409336] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.413836] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.419409] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.423976] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.430655] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.437752] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Switch Root. Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice User and Session Slice. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. Mounting Kernel Debug File System... [ OK ] Created slice system-getty.slice. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting Huge Pages File System... Starting Apply Kernel Variables... Activating swap /dev/disk/by-label/SWAP... [ OK ] Stopped target Initrd File Systems. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on initctl Compatibility Named Pipe. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ 12.573829] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK ] Started udev Coldplug all Devices. [ 12.879082] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.085152] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.122981] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.257751] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.271317] EDAC sbridge: Ver: 1.1.2 [ 15.001222] Key type dns_resolver registered [ 15.304387] NFS: Registering the id_resolver key type [ 15.306885] Key type id_resolver registered [ 15.308585] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting Restore /run/initramfs on shutdown... Starting Login Service... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started OpenSSH server daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg355-server login: [ 43.804022] hrtimer: interrupt took 3883530 ns [ 47.651735] libcfs: loading out-of-tree module taints kernel. [ 47.694804] Key type ._llcrypt registered [ 47.697611] Key type .llcrypt registered [ 47.795828] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 64.811493] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 66.004252] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 66.017656] alg: No test for adler32 (adler32-zlib) [ 67.382912] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 68.190646] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 69.919174] Key type lgssc registered [ 71.579707] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.333102] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 127.970743] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 139.620240] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 139.676480] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 140.950947] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 140.984754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 141.076675] Lustre: lustre-MDT0000: new disk, initializing [ 141.165338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 141.194140] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 145.268226] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 158.622216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 158.759042] Lustre: 6515:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 158.810300] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 158.815615] Lustre: Skipped 1 previous similar message [ 158.905823] Lustre: lustre-MDT0001: new disk, initializing [ 158.977770] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 159.011887] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 159.019895] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 162.999274] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 167.739606] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 176.756436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 177.037821] Lustre: lustre-OST0000: new disk, initializing [ 177.045893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 177.055858] Lustre: 8454:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 177.149637] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 184.186781] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 185.893827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 185.902696] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 185.937338] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 197.332792] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 197.476997] Lustre: lustre-OST0001: new disk, initializing [ 197.484603] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 197.493861] Lustre: 9525:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 197.557701] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 203.275120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 203.280767] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 203.336708] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 203.776253] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 216.492205] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 226.040477] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 233.033150] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing check_logdir /tmp/testlogs/ [ 238.355287] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing yml_node [ 242.667841] Lustre: DEBUG MARKER: Client: 2.17.58.3 [ 245.277799] Lustre: DEBUG MARKER: MDS: 2.17.58.3 [ 247.995089] Lustre: DEBUG MARKER: OSS: 2.17.58.3 [ 249.677718] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Sat Sep 5 17:01:11 EDT 2026 [ 266.329916] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 267.856896] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 285.152398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 285.155482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 285.172211] Lustre: Skipped 3 previous similar messages [ 285.190054] Lustre: Skipped 3 previous similar messages [ 289.385650] Lustre: server umount lustre-MDT0000 complete [ 293.025503] LustreError: 7463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642117 with bad export cookie 10356734745688639149 [ 293.030125] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 293.046299] LustreError: 7463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 295.396269] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 295.403786] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 295.404480] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 295.410023] Lustre: Skipped 1 previous similar message [ 295.445666] LustreError: 6524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 298.401305] LustreError: 9531:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 298.401803] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 298.425490] Lustre: Skipped 2 previous similar messages [ 299.441891] Lustre: server umount lustre-MDT0001 complete [ 303.176499] Lustre: server umount lustre-OST0000 complete [ 306.672595] Lustre: server umount lustre-OST0001 complete [ 312.961825] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 324.086859] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 361.572477] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 361.824941] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 361.851799] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 361.924687] Lustre: lustre-MDT0000: new disk, initializing [ 362.006431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 362.026626] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 366.477906] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 377.809045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 377.943523] Lustre: 15090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 378.007793] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 378.012529] Lustre: Skipped 1 previous similar message [ 378.127573] Lustre: lustre-MDT0001: new disk, initializing [ 378.208846] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 378.256136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 378.268145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 382.669754] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 391.872750] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 397.421657] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 403.905250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 404.163893] Lustre: lustre-OST0000: new disk, initializing [ 404.167237] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 404.171677] Lustre: 17058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 404.249296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 405.638034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 405.651376] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 405.781832] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 410.469402] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 419.848478] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 425.269543] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 425.573765] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 430.667562] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 430.965581] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 436.710586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 436.720044] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 436.727962] Lustre: Skipped 1 previous similar message [ 436.731128] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 439.128220] Lustre: server umount lustre-OST0000 complete [ 444.896315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 444.914469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 444.936842] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 444.946671] Lustre: Skipped 1 previous similar message [ 449.392586] Lustre: server umount lustre-MDT0000 complete [ 452.072555] LustreError: 15100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 452.100964] LustreError: 15100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 453.060256] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642277 with bad export cookie 10356734745688640689 [ 453.069624] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 453.073776] LustreError: 15082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 453.509940] Lustre: server umount lustre-MDT0001 complete [ 458.042448] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 17:04:40 (1788642280) [ 460.202702] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 461.826975] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 17:04:44 (1788642284) [ 468.388136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 468.821761] LustreError: 19467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 468.924857] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 473.932234] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 474.081568] LustreError: 19468:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.917307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 486.993862] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 494.694664] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 500.456312] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 506.920121] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 513.604207] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 521.218813] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 528.864328] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 528.874295] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 528.887786] Lustre: Skipped 1 previous similar message [ 528.900944] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.904022] Lustre: Skipped 2 previous similar messages [ 534.333461] Lustre: server umount lustre-OST0000 complete [ 538.592138] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 538.614539] LustreError: Skipped 1 previous similar message [ 538.633246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 538.656144] Lustre: Skipped 1 previous similar message [ 544.579019] Lustre: server umount lustre-MDT0000 complete [ 545.255408] LustreError: 20901:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.274326] LustreError: 20901:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 548.284320] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642373 with bad export cookie 10356734745688641571 [ 548.291218] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 548.298695] LustreError: 19449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 548.677145] Lustre: server umount lustre-MDT0001 complete [ 560.483272] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 563.415301] Key type lgssc unregistered [ 563.830393] LNet: 22999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 563.839370] LNetError: 22999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 563.851772] LNet: Removed LNI 192.168.203.155@tcp [ 564.828151] Key type .llcrypt unregistered [ 564.833820] Key type ._llcrypt unregistered [ 588.768379] Key type ._llcrypt registered [ 588.770670] Key type .llcrypt registered [ 588.936624] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 590.407187] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 590.466147] alg: No test for adler32 (adler32-zlib) [ 591.633309] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 591.874693] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 593.599145] Key type lgssc registered [ 594.797666] Lustre: Echo OBD driver; http://www.lustre.org/ [ 603.394032] loop: module loaded [ 603.486216] loop0: detected capacity change from 0 to 204800000 [ 603.810677] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 603.842537] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 604.164232] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 611.207630] loop0: detected capacity change from 0 to 204800000 [ 611.439725] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 611.453598] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 611.470189] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 617.373872] loop0: detected capacity change from 0 to 204800000 [ 623.034399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 623.053447] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 624.419151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 624.455355] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 624.544262] Lustre: lustre-MDT0000: new disk, initializing [ 624.645721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 624.686329] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 629.062054] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 638.094772] loop1: detected capacity change from 0 to 204800000 [ 643.761282] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 644.076779] Lustre: lustre-OST07c6: new disk, initializing [ 644.082645] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 644.087810] Lustre: Skipped 1 previous similar message [ 644.092692] Lustre: 26184:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 644.165198] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 650.052948] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 653.954629] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 653.967552] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 654.011713] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 655.705825] Lustre: 25171:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 655.718444] Lustre: 25171:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 655.732520] Lustre: 25171:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 655.743938] Lustre: 25171:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 655.749975] Lustre: 25171:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 655.756935] Lustre: 25171:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 674.213927] Lustre: 26816:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 674.219419] Lustre: 26816:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 674.228853] Lustre: 26816:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 674.238127] Lustre: 26816:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 674.249107] Lustre: 26816:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 674.258394] Lustre: 26816:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 679.435731] Lustre: 26816:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 679.443041] Lustre: 26816:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 679.447477] Lustre: 26816:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 679.451596] Lustre: 26816:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.457376] Lustre: 26816:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 679.463974] Lustre: 26816:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.469777] Lustre: 26816:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 679.477995] Lustre: 26816:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.489392] Lustre: 26816:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 679.498411] Lustre: 26816:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 679.504301] Lustre: 26816:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 679.512021] Lustre: 26816:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.766524] Lustre: 25170:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 682.778300] Lustre: 25170:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 682.798445] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 682.808910] Lustre: 25170:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.817159] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 682.822658] Lustre: 25170:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.829852] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 682.839956] Lustre: 25170:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.849422] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 682.855419] Lustre: 25170:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 682.861320] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 682.866635] Lustre: 25170:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.730281] Lustre: 25169:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 687.749746] Lustre: 25169:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 687.767097] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 687.783365] Lustre: 25169:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.790434] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 687.805455] Lustre: 25169:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.819885] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 687.825705] Lustre: 25169:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.830906] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 687.839705] Lustre: 25169:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 687.848336] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 687.854579] Lustre: 25169:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.378175] Lustre: 26816:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 696.393680] Lustre: 26816:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 696.403372] Lustre: 26816:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 696.415830] Lustre: 26816:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 696.425174] Lustre: 26816:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 696.445484] Lustre: 26816:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 696.451398] Lustre: 26816:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 696.460620] Lustre: 26816:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 696.467226] Lustre: 26816:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 696.480552] Lustre: 26816:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 696.485974] Lustre: 26816:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 696.496480] Lustre: 26816:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 751.076098] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 751.089076] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 751.110114] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 754.883667] Lustre: server umount lustre-OST07c6 complete [ 764.840412] Lustre: server umount lustre-MDT0000 complete [ 781.769538] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 784.435119] Key type lgssc unregistered [ 784.751814] LNet: 28741:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 784.758237] LNetError: 28741:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 784.770778] LNet: Removed LNI 192.168.203.155@tcp [ 785.700467] Key type .llcrypt unregistered [ 785.703449] Key type ._llcrypt unregistered [ 810.526366] Key type ._llcrypt registered [ 810.532149] Key type .llcrypt registered [ 810.685712] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 812.125729] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 812.300787] alg: No test for adler32 (adler32-zlib) [ 813.337353] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 813.733215] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 815.504947] Key type lgssc registered [ 816.954797] Lustre: Echo OBD driver; http://www.lustre.org/ [ 824.396967] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 824.415663] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 825.895033] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 830.150780] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 836.072209] LustreError: 30012:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 836.109591] LustreError: 30012:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 839.073978] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 839.443707] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 843.997171] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 851.816352] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 857.416463] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 865.403378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 865.863692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 871.047889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 873.506195] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 882.051513] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 900.072775] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 900.075322] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.090981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 900.097650] Lustre: Skipped 1 previous similar message [ 900.120809] Lustre: Skipped 1 previous similar message [ 903.724739] Lustre: server umount lustre-OST0000 complete [ 910.305130] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 910.331065] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 910.345155] Lustre: Skipped 1 previous similar message [ 914.257248] Lustre: server umount lustre-MDT0000 complete [ 915.427464] LustreError: 32706:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 915.452821] LustreError: 32706:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 917.954280] LustreError: 29993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642742 with bad export cookie 17679814565892354757 [ 917.970431] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 918.309938] Lustre: server umount lustre-MDT0001 complete [ 928.500523] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 931.216892] Key type lgssc unregistered [ 931.614539] LNet: 33747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 931.626497] LNetError: 33747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 931.655398] LNet: Removed LNI 192.168.203.155@tcp [ 932.701415] Key type .llcrypt unregistered [ 932.703440] Key type ._llcrypt unregistered [ 950.461378] Key type ._llcrypt registered [ 950.464475] Key type .llcrypt registered [ 950.554860] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 951.604716] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 951.629973] alg: No test for adler32 (adler32-zlib) [ 952.687411] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 952.916691] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 954.631454] Key type lgssc registered [ 955.919552] Lustre: Echo OBD driver; http://www.lustre.org/ [ 963.397527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 963.436432] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 965.077628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 969.584769] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 974.307640] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 974.342679] LustreError: 35010:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 977.916702] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 978.230414] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 982.441381] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 990.018678] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 995.463059] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1002.334019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1002.615371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1008.929712] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1009.840136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1016.293899] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1029.602268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1029.618889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.643521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1030.624591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1030.625697] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1030.653932] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1032.603358] Lustre: server umount lustre-OST0000 complete [ 1039.842340] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1039.855354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1039.888280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1039.903234] Lustre: Skipped 1 previous similar message [ 1043.190572] Lustre: server umount lustre-MDT0000 complete [ 1045.986033] LustreError: 35004:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1046.010216] LustreError: 35004:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1047.300425] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788642872 with bad export cookie 1922318242079031467 [ 1047.306298] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1047.321365] LustreError: 34990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1047.743760] Lustre: server umount lustre-MDT0001 complete [ 1058.105346] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1060.646027] Key type lgssc unregistered [ 1060.926740] LNet: 38540:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1060.939948] LNetError: 38540:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1060.967162] LNet: Removed LNI 192.168.203.155@tcp [ 1061.720891] Key type .llcrypt unregistered [ 1061.723183] Key type ._llcrypt unregistered [ 1085.311592] Key type ._llcrypt registered [ 1085.313833] Key type .llcrypt registered [ 1085.436024] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1086.549096] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1086.737262] alg: No test for adler32 (adler32-zlib) [ 1087.869523] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 1088.198486] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1089.887206] Key type lgssc registered [ 1090.991669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1098.465618] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1098.515533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1100.025883] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.424246] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1109.481773] LustreError: 39956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1109.516828] LustreError: 39956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1113.031467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1113.436531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1118.474683] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1126.551650] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1132.730724] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1139.671484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1139.998182] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1145.135103] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1146.682707] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1154.762742] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1167.869821] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1168.869456] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1168.874553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1168.886425] Lustre: Skipped 2 previous similar messages [ 1168.914797] Lustre: Skipped 2 previous similar messages [ 1171.532967] Lustre: server umount lustre-MDT0000 complete [ 1172.948051] LustreError: 39952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1173.994087] LustreError: 39951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1174.024671] LustreError: 39951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1175.608830] LustreError: 39936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643000 with bad export cookie 18402310609027330162 [ 1175.632725] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1178.066468] LustreError: 39952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1178.126550] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 1179.107574] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1180.139596] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1180.153630] Lustre: Skipped 1 previous similar message [ 1182.201911] Lustre: server umount lustre-MDT0001 complete [ 1198.441772] Lustre: server umount lustre-OST0000 complete [ 1219.341618] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1222.451523] Key type lgssc unregistered [ 1222.738797] LNet: 43818:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1222.745981] LNetError: 43818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1222.764952] LNet: Removed LNI 192.168.203.155@tcp [ 1223.660869] Key type .llcrypt unregistered [ 1223.664734] Key type ._llcrypt unregistered [ 1248.505611] Key type ._llcrypt registered [ 1248.513225] Key type .llcrypt registered [ 1248.663364] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1249.835987] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1249.880167] alg: No test for adler32 (adler32-zlib) [ 1251.171320] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 1251.454090] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1253.231206] Key type lgssc registered [ 1254.360836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1262.140561] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1262.183354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1263.722094] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1269.482804] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1273.328685] LustreError: 45234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1273.378252] LustreError: 45234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1278.435336] LustreError: 45235:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1279.310415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1279.823685] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1285.633955] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1294.723437] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1302.101677] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1309.671697] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1310.030791] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1316.663062] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1325.556684] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1333.619080] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1342.458187] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1343.982223] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1343.989099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1343.992179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1343.997041] Lustre: Skipped 1 previous similar message [ 1344.006807] Lustre: Skipped 1 previous similar message [ 1347.545038] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1348.449628] Lustre: server umount lustre-OST0000 complete [ 1367.010332] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1367.030285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1367.055635] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1368.555439] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1373.665251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1373.672348] Lustre: Skipped 3 previous similar messages [ 1381.343142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1381.503865] Lustre: server umount lustre-MDT0000 complete [ 1383.916546] LustreError: 46525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1383.934010] LustreError: 46525:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1384.993356] LustreError: 45214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643209 with bad export cookie 6533080329295630724 [ 1385.000280] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1385.008842] LustreError: 45214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1391.394637] Lustre: server umount lustre-MDT0001 complete [ 1404.244626] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1407.143425] Key type lgssc unregistered [ 1407.392126] LNet: 48966:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.400839] LNetError: 48966:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1407.414668] LNet: Removed LNI 192.168.203.155@tcp [ 1408.316131] Key type .llcrypt unregistered [ 1408.318640] Key type ._llcrypt unregistered [ 1432.182376] Key type ._llcrypt registered [ 1432.184114] Key type .llcrypt registered [ 1432.281164] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1433.809132] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1433.841202] alg: No test for adler32 (adler32-zlib) [ 1434.883200] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 1435.203226] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1436.879193] Key type lgssc registered [ 1438.024614] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1445.209526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1445.226370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1446.705653] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1451.229982] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1457.132405] LustreError: 50376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1457.173201] LustreError: 50376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1460.041293] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1460.445160] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1465.626770] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1473.920309] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1479.372937] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1485.992744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1486.449304] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1491.582090] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1492.908991] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1499.957104] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1509.271571] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1512.519514] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1524.501416] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0517h22m26s [ 1526.208649] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1528.307408] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1529.359921] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 1531.881640] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1531.885103] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1531.889809] Lustre: Skipped 1 previous similar message [ 1534.420240] LustreError: 50372:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1534.423176] Lustre: server umount lustre-MDT0000 complete [ 1537.388741] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1539.048356] LustreError: 53062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1539.067368] LustreError: 53062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1544.171713] LustreError: 50373:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1544.186701] LustreError: 50373:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1544.206109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1544.344615] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1544.621677] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1544.703581] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1545.650902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1549.547099] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1553.172563] Lustre: DEBUG MARKER: Wait for df (36679) ... [ 1554.824663] Lustre: DEBUG MARKER: done [ 1560.035821] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1560.051429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1560.077736] Lustre: Skipped 1 previous similar message [ 1560.086516] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1560.090700] Lustre: Skipped 5 previous similar messages [ 1561.059277] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1565.452110] Lustre: server umount lustre-OST0000 complete [ 1571.296878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1571.317711] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1571.324930] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1571.325200] Lustre: Skipped 1 previous similar message [ 1571.362848] Lustre: Skipped 3 previous similar messages [ 1575.116313] Lustre: server umount lustre-MDT0000 complete [ 1578.470075] LustreError: 52215:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1578.487583] LustreError: 52215:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1578.597352] LustreError: 50358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643403 with bad export cookie 17258735134485508482 [ 1578.598325] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1578.608479] LustreError: 50358:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1578.923588] Lustre: server umount lustre-MDT0001 complete [ 1590.982843] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1593.757055] Key type lgssc unregistered [ 1594.129230] LNet: 55296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1594.137297] LNetError: 55296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1594.150411] LNet: Removed LNI 192.168.203.155@tcp [ 1595.098636] Key type .llcrypt unregistered [ 1595.101284] Key type ._llcrypt unregistered [ 1638.273029] Key type ._llcrypt registered [ 1638.281202] Key type .llcrypt registered [ 1638.437645] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1639.419413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1639.571149] alg: No test for adler32 (adler32-zlib) [ 1640.724805] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 1640.995505] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1642.801906] Key type lgssc registered [ 1643.917629] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1651.862349] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1651.878149] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1653.051146] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1653.076308] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1653.233556] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1657.161877] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1665.196749] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1665.292899] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1665.321581] Lustre: 57499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1665.537155] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1670.071092] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1679.517431] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1687.357637] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1695.210298] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1695.471793] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1695.766839] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1701.907736] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1703.089619] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1710.444931] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1717.694287] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0517h25m39s [ 1719.899207] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1727.895306] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1730.540687] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1730.595741] LustreError: 57507:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1751.055052] Lustre: lustre-MDT0000: Client 70b2bbc8-05c3-487e-87bf-1ed97bf694ec (at 192.168.203.55@tcp) reconnecting [ 1770.609575] LustreError: 57507:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1773.594590] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1776.286265] Lustre: DEBUG MARKER: done [ 1781.575989] Lustre: server umount lustre-OST0000 complete [ 1788.386036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1788.394696] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1788.406916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1789.474096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1789.476876] Lustre: Skipped 1 previous similar message [ 1795.552993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1795.569156] Lustre: Skipped 1 previous similar message [ 1795.657216] Lustre: server umount lustre-MDT0000 complete [ 1799.743149] LustreError: 57490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643624 with bad export cookie 15202492463741278553 [ 1799.743946] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1799.753406] LustreError: 57490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1809.501047] Lustre: server umount lustre-MDT0001 complete [ 1820.614376] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 1823.260497] Key type lgssc unregistered [ 1823.590337] LNet: 61831:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1823.599864] LNetError: 61831:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1823.615689] LNet: Removed LNI 192.168.203.155@tcp [ 1824.514554] Key type .llcrypt unregistered [ 1824.516578] Key type ._llcrypt unregistered [ 1868.487372] Key type ._llcrypt registered [ 1868.490301] Key type .llcrypt registered [ 1868.563465] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 1869.362709] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1869.379065] alg: No test for adler32 (adler32-zlib) [ 1870.403486] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 1870.576692] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 1872.233923] Key type lgssc registered [ 1873.447728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1876.270553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1876.295750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1877.568974] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1877.594997] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1877.801053] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1878.801580] Lustre: Failing over lustre-MDT0000 [ 1879.059705] Lustre: server umount lustre-MDT0000 complete [ 1886.344103] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 17:28:28 (1788643708) [ 1892.483269] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1892.885501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1897.796536] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1906.396635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1906.524838] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1906.548759] Lustre: 64623:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1906.742284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1910.046415] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1917.259534] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1924.525331] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1932.185781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1932.293430] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1932.466234] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1941.158151] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 1941.722329] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1950.675689] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1958.108924] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1967.586177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1967.592881] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1967.601703] Lustre: Skipped 1 previous similar message [ 1972.715146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1972.724877] Lustre: Skipped 4 previous similar messages [ 1977.838247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1977.849655] Lustre: Skipped 2 previous similar messages [ 1979.359339] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1979.758384] Lustre: server umount lustre-MDT0000 complete [ 1983.762749] LustreError: 64614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643808 with bad export cookie 11099827402698491071 [ 1983.772191] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1983.776684] LustreError: 64614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1984.149917] Lustre: server umount lustre-MDT0001 complete [ 1987.560651] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1991.818447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2000.114486] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2003.423140] Lustre: 63546:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788643812/real 1788643812] req@000000008f1976d8 x1875528851171840/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788643828 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2003.456884] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.471782] Lustre: Skipped 1 previous similar message [ 2008.543588] Lustre: 63547:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788643817/real 1788643817] req@00000000cc0ba2fd x1875528851171968/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788643833 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2009.934117] LustreError: 66470:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2009.956278] LustreError: 66470:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2010.007647] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2010.094326] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2010.979096] Lustre: 63545:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788643819/real 1788643819] req@000000000aadd356 x1875528851172224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788643835 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2013.089787] LustreError: 66471:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2013.098521] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2014.125486] LustreError: 68393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2014.144790] LustreError: 68393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2014.347923] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2016.170703] Lustre: 63545:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788643824/real 1788643824] req@00000000f64d40d9 x1875528851172352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788643840 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2019.302978] LustreError: 68393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2019.310825] LustreError: 68393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2024.435579] LustreError: 68391:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.489704] LustreError: 68391:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2024.526885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2025.405819] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2030.583386] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2031.439390] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2040.079462] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2047.157148] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2056.162388] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2056.168873] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2056.176123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2056.185597] Lustre: Skipped 2 previous similar messages [ 2056.202390] Lustre: Skipped 2 previous similar messages [ 2061.287121] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2061.294356] Lustre: Skipped 2 previous similar messages [ 2061.831695] Lustre: server umount lustre-MDT0000 complete [ 2065.728934] LustreError: 64616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643890 with bad export cookie 11099827402698493353 [ 2065.730133] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2065.737681] LustreError: 64616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2066.234994] Lustre: server umount lustre-MDT0001 complete [ 2073.216132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2082.783847] Lustre: 63546:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788643891/real 1788643891] req@000000002babe977 x1875528851200896/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788643907 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2082.818061] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2084.704440] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2091.040269] LustreError: 63543:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f94a1cda x1875528851201920/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2091.377180] LustreError: 69129:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2091.497760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2091.600026] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2092.522109] Lustre: 63545:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788643900/real 1788643900] req@000000005e15409e x1875528851201408/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788643916 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2092.566240] Lustre: 63545:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 2096.342990] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2105.233318] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2110.079569] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2110.953578] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2118.103374] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2124.669679] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2132.451294] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2132.455122] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2132.480672] Lustre: Skipped 1 previous similar message [ 2146.272316] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2146.461961] Lustre: server umount lustre-MDT0000 complete [ 2147.809888] LustreError: 71812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.844191] LustreError: 71812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 2149.669673] LustreError: 65366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788643974 with bad export cookie 11099827402698494732 [ 2149.670284] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2149.677646] LustreError: 65366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2149.929128] Lustre: server umount lustre-MDT0001 complete [ 2156.664720] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2160.488645] Lustre: DEBUG MARKER: files compared the same [ 2172.977109] Lustre: server umount lustre-OST0000 complete [ 2190.649619] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2193.589620] Key type lgssc unregistered [ 2193.841385] LNet: 74355:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2193.852751] LNetError: 74355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2193.870301] LNet: Removed LNI 192.168.203.155@tcp [ 2194.780394] Key type .llcrypt unregistered [ 2194.783609] Key type ._llcrypt unregistered [ 2222.677918] Key type ._llcrypt registered [ 2222.681540] Key type .llcrypt registered [ 2222.806507] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2224.168082] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2224.312136] alg: No test for adler32 (adler32-zlib) [ 2225.466688] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 2225.793145] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2227.583207] Key type lgssc registered [ 2228.937639] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2237.933901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2237.964785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2239.380900] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2244.588845] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2249.697114] LustreError: 75768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2249.714134] LustreError: 75768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2253.560977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2253.880416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2259.320048] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2267.806763] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2274.791887] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2282.427153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2282.822778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2284.865524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2289.951770] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug +malloc all [ 2298.648074] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2309.600853] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2309.610586] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2309.614572] Lustre: Skipped 1 previous similar message [ 2314.043950] Lustre: server umount lustre-OST0000 complete [ 2319.840117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2319.862129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2319.875401] Lustre: Skipped 2 previous similar messages [ 2324.231183] Lustre: server umount lustre-MDT0000 complete [ 2324.965588] LustreError: 75768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.990814] LustreError: 75768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2327.639277] LustreError: 75748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644152 with bad export cookie 9330789304066493063 [ 2327.642207] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2327.649113] LustreError: 75748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2327.945371] Lustre: server umount lustre-MDT0001 complete [ 2339.570745] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2342.815557] Key type lgssc unregistered [ 2343.170633] LNet: 79296:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2343.179166] LNetError: 79296:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2343.192617] LNet: Removed LNI 192.168.203.155@tcp [ 2344.263909] Key type .llcrypt unregistered [ 2344.265625] Key type ._llcrypt unregistered [ 2371.889216] Key type ._llcrypt registered [ 2371.892491] Key type .llcrypt registered [ 2372.038455] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2373.426020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2373.488159] alg: No test for adler32 (adler32-zlib) [ 2374.608412] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 2374.951645] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2376.687891] Key type lgssc registered [ 2377.826281] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2386.085601] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2386.114019] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2401.824972] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2406.944157] LustreError: 80685:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [ 2433.567417] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2441.341331] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2451.553445] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2459.559272] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2459.708286] Lustre: *** cfs_fail_loc=706, val=0*** [ 2460.110182] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2460.189499] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2464.717472] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2465.254775] LustreError: 80711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2465.268900] LustreError: 80711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2468.325397] LustreError: 80712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2468.341619] LustreError: 80712:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2473.448233] LustreError: 80711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2473.472406] LustreError: 80711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2473.755759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2474.276978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2479.264259] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2487.335292] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2493.020150] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2495.983076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2495.989898] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2496.005299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2500.073560] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2500.087425] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2500.096369] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2500.122813] Lustre: Skipped 1 previous similar message [ 2505.190828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2505.201893] Lustre: Skipped 1 previous similar message [ 2509.279121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2509.442916] Lustre: server umount lustre-OST0000 complete [ 2515.223234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2515.232300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2520.547828] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2520.550957] Lustre: Skipped 2 previous similar messages [ 2521.777193] Lustre: server umount lustre-MDT0000 complete [ 2525.670876] LustreError: 81772:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2525.710240] LustreError: 81772:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2526.933602] LustreError: 80693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644351 with bad export cookie 15958780095908371896 [ 2526.940858] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2526.941544] LustreError: 80693:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2527.138219] Lustre: server umount lustre-MDT0001 complete [ 2543.081061] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2547.626964] Key type lgssc unregistered [ 2548.063485] LNet: 84308:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.080249] LNetError: 84308:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.107825] LNet: Removed LNI 192.168.203.155@tcp [ 2549.566316] Key type .llcrypt unregistered [ 2549.568869] Key type ._llcrypt unregistered [ 2578.023886] Key type ._llcrypt registered [ 2578.026015] Key type .llcrypt registered [ 2578.138243] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2579.196445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2579.255181] alg: No test for adler32 (adler32-zlib) [ 2580.288584] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 2580.545780] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2582.199760] Key type lgssc registered [ 2583.387746] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2591.151514] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2591.174618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2596.360875] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2604.397180] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2605.080942] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2609.393377] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2610.145924] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2610.170755] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2614.240400] LustreError: 86412:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2614.258082] LustreError: 86412:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2617.945040] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2618.255471] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2618.286864] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2618.303311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2623.460127] LustreError: 86407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2623.497874] LustreError: 86407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2624.916989] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2628.583736] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2628.600209] LustreError: 86408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2634.497954] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2634.510313] Lustre: MGS: Client bf74c18c-572a-4638-ac2a-0994fe7a9618 (at 0@lo) reconnecting [ 2634.530743] LustreError: 86422:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2634.785625] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2634.889580] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2639.728918] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2655.214084] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2655.214374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2655.225699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2655.262911] Lustre: Skipped 1 previous similar message [ 2659.417721] Lustre: server umount lustre-OST0000 complete [ 2665.443344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2665.444515] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2665.457409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2665.475113] Lustre: Skipped 2 previous similar messages [ 2665.493080] Lustre: Skipped 1 previous similar message [ 2670.560770] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2670.567619] Lustre: Skipped 1 previous similar message [ 2675.690385] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2675.698293] Lustre: Skipped 1 previous similar message [ 2677.215226] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2677.329816] Lustre: server umount lustre-MDT0000 complete [ 2680.810630] LustreError: 87953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2680.834826] LustreError: 87953:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2681.598048] Lustre: server umount lustre-MDT0001 complete [ 2685.231180] Lustre: server umount MGS complete [ 2697.839870] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 17:41:59 (1788644519) [ 2730.496327] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 2740.851940] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2777.856507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2782.971348] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2794.211162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2794.399168] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2794.458171] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2794.575581] Lustre: lustre-MDT0001: new disk, initializing [ 2794.752739] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2794.844854] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2794.860760] Lustre: 93591:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2794.876422] LustreError: 93591:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2799.352623] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2810.445598] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2810.846710] Lustre: lustre-OST0000: new disk, initializing [ 2810.853213] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2810.861197] Lustre: 94358:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2810.964968] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2817.532137] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2822.243366] Lustre: 94842:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000414d2576 x1875529595659008/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788644663 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2828.201605] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2835.735060] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2835.753857] Lustre: MGS: Client 01f41aa5-18e6-4626-a828-0dc0282b8fcf (at 0@lo) reconnecting [ 2835.795580] Lustre: 92858:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2835.824463] LustreError: 92858:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x0 : rc = -11 [ 2835.831727] LustreError: 92858:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2835.851914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2835.924593] Lustre: lustre-MDT0000: new disk, initializing [ 2836.048950] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2836.073661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2836.166707] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2837.096427] Lustre: 95426:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000066e41b28 x1875529595665920/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788644677 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2840.781530] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2842.593118] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2846.193348] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2846.205840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2846.358696] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2872.288875] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2872.306401] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2872.318483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2874.855246] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2876.669240] Lustre: server umount lustre-OST0000 complete [ 2885.089087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2885.108358] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2885.113979] Lustre: Skipped 3 previous similar messages [ 2886.430634] Lustre: server umount lustre-MDT0000 complete [ 2896.408114] Lustre: server umount lustre-MDT0001 complete [ 2900.313965] Lustre: server umount MGS complete [ 2911.340252] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 17:45:33 (1788644733) [ 2931.215602] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 2933.819783] Key type lgssc unregistered [ 2934.118865] LNet: 97921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2934.124691] LNetError: 97921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2934.137751] LNet: Removed LNI 192.168.203.155@tcp [ 2934.699892] Key type .llcrypt unregistered [ 2934.704080] Key type ._llcrypt unregistered [ 2953.132478] Key type ._llcrypt registered [ 2953.134874] Key type .llcrypt registered [ 2953.239954] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 2954.144967] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2954.232786] alg: No test for adler32 (adler32-zlib) [ 2955.308620] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 2955.668112] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 2957.391346] Key type lgssc registered [ 2959.173343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2969.621409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2969.640015] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2970.998148] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2974.769724] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2980.322588] LustreError: 99137:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2980.339896] LustreError: 99137:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2981.786883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2982.046728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2984.899360] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 2992.560649] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2998.908129] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3005.694042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3006.060689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3012.603309] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3015.358201] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3020.715572] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3031.009108] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3031.021701] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3031.025491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3031.042607] Lustre: Skipped 1 previous similar message [ 3031.056815] Lustre: Skipped 1 previous similar message [ 3034.454731] Lustre: server umount lustre-OST0000 complete [ 3041.256814] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3041.263432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3041.268631] Lustre: Skipped 1 previous similar message [ 3041.282571] Lustre: Skipped 1 previous similar message [ 3044.532965] Lustre: server umount lustre-MDT0000 complete [ 3046.373796] LustreError: 99870:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3046.389183] LustreError: 99870:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3047.817365] LustreError: 99119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644872 with bad export cookie 15246680549069043415 [ 3047.825426] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3047.831617] LustreError: 99119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3048.089161] Lustre: server umount lustre-MDT0001 complete [ 3059.870353] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3062.530248] Key type lgssc unregistered [ 3062.801562] LNet: 102669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3062.809159] LNetError: 102669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3062.824315] LNet: Removed LNI 192.168.203.155@tcp [ 3063.703987] Key type .llcrypt unregistered [ 3063.711271] Key type ._llcrypt unregistered [ 3081.123332] Key type ._llcrypt registered [ 3081.130446] Key type .llcrypt registered [ 3081.210415] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3082.220504] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3082.244719] alg: No test for adler32 (adler32-zlib) [ 3083.310854] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 3083.523931] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3085.191471] Key type lgssc registered [ 3086.347131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3092.028987] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3092.040826] LustreError: 103744:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3092.068205] LustreError: 103749:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3092.073902] LustreError: 103744:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4979 [ 3092.082611] LustreError: 103744:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3092.091785] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.100975] LustreError: 103744:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3093.540389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3101.854844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3102.186189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3107.062516] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3112.124359] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lsmod [ 3114.960843] LustreError: 104956:0:(tgt_mount.c:2573:server_fill_super()) cfs_race id 716 sleeping [ 3114.976689] LustreError: 104957:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 waking [ 3114.989083] LustreError: 104956:0:(tgt_mount.c:2573:server_fill_super()) cfs_fail_race id 716 awake: rc=4976 [ 3115.000212] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3115.000757] LustreError: 104956:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3115.000795] LustreError: 104956:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3115.565639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3120.608193] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3120.623983] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3120.628523] Lustre: Skipped 2 previous similar messages [ 3120.646745] Lustre: Skipped 2 previous similar messages [ 3124.560306] Lustre: server umount lustre-MDT0000 complete [ 3125.731185] LustreError: 103783:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3125.761815] LustreError: 103783:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3126.755506] LustreError: 103788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3126.800443] LustreError: 103788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3128.195622] LustreError: 103766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788644952 with bad export cookie 16935705107877870584 [ 3128.204756] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3128.216391] LustreError: 103766:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3128.839282] LustreError: 104219:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000e67fe0ce x1875530123071872/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3128.976425] Lustre: server umount lustre-MDT0001 complete [ 3141.643067] Lustre: server umount lustre-OST0000 complete [ 3150.527294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3150.928257] LustreError: 106022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3150.985786] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3155.574279] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3156.449851] LustreError: 106023:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.553879] LustreError: 106022:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3163.780483] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3164.184382] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3168.424625] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3176.490484] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3182.307974] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3189.309477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3189.848486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3195.534534] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3196.022905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3201.697251] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3218.913353] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3218.915735] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3218.948086] Lustre: Skipped 1 previous similar message [ 3218.973449] Lustre: Skipped 1 previous similar message [ 3224.033848] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3224.051150] Lustre: Skipped 1 previous similar message [ 3225.447775] Lustre: server umount lustre-OST0000 complete [ 3229.156101] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3229.177118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3229.191402] Lustre: Skipped 1 previous similar message [ 3234.272679] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3234.280164] Lustre: Skipped 1 previous similar message [ 3235.180915] Lustre: server umount lustre-MDT0000 complete [ 3238.466019] LustreError: 106756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645063 with bad export cookie 16935705107877871298 [ 3238.469958] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3238.480381] LustreError: 106756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3238.855551] Lustre: server umount lustre-MDT0001 complete [ 3248.717082] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3251.123756] Key type lgssc unregistered [ 3251.376694] LNet: 109572:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3251.385185] LNetError: 109572:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3251.402801] LNet: Removed LNI 192.168.203.155@tcp [ 3252.123106] Key type .llcrypt unregistered [ 3252.126408] Key type ._llcrypt unregistered [ 3273.458550] Key type ._llcrypt registered [ 3273.461312] Key type .llcrypt registered [ 3273.571747] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3274.543391] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3274.567980] alg: No test for adler32 (adler32-zlib) [ 3275.722105] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 3275.969142] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3277.623256] Key type lgssc registered [ 3278.727773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3287.136117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3287.180377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3288.667508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3293.114841] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3298.787571] LustreError: 110987:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3298.819729] LustreError: 110987:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3302.098407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3302.421968] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3307.295412] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3314.561571] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3319.874276] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3326.906868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3327.378163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3333.563855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3334.068890] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3341.411166] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3346.212348] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3349.875934] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3353.861534] Lustre: server umount lustre-OST0000 complete [ 3358.697182] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3358.705304] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3358.717211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3364.321966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3364.329862] Lustre: Skipped 3 previous similar messages [ 3367.506695] Lustre: server umount lustre-MDT0000 complete [ 3369.443803] LustreError: 110983:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3369.480198] LustreError: 110983:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3369.482214] Lustre: 110298:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788645178/real 1788645178] req@000000001cd33186 x1875530324406784/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788645194 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3369.534359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3369.560112] Lustre: Skipped 1 previous similar message [ 3371.133801] LustreError: 112831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645195 with bad export cookie 6653835786785718271 [ 3371.144443] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3371.149934] LustreError: 112831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3371.505355] Lustre: server umount lustre-MDT0001 complete [ 3381.865405] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3384.482970] Key type lgssc unregistered [ 3384.774161] LNet: 114624:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3384.779359] LNetError: 114624:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3384.797159] LNet: Removed LNI 192.168.203.155@tcp [ 3385.573709] Key type .llcrypt unregistered [ 3385.576951] Key type ._llcrypt unregistered [ 3405.869650] Key type ._llcrypt registered [ 3405.873768] Key type .llcrypt registered [ 3405.983879] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3406.710436] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3406.720945] alg: No test for adler32 (adler32-zlib) [ 3407.693632] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 3407.964470] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3409.679193] Key type lgssc registered [ 3410.605689] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3417.884426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3417.895498] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3419.341672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.921834] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3429.856164] LustreError: 115842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3429.881980] LustreError: 115842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3432.305157] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.734645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3437.410137] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3446.550045] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3454.867174] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3463.093680] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3463.520811] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3465.595724] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3470.645416] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3479.349416] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3488.225513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3488.241180] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3488.243789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3488.256435] Lustre: Skipped 1 previous similar message [ 3493.345276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3493.352557] Lustre: Skipped 1 previous similar message [ 3494.188962] Lustre: server umount lustre-OST0000 complete [ 3498.468310] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3498.488827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3498.496176] Lustre: Skipped 1 previous similar message [ 3503.584974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3503.601295] Lustre: Skipped 1 previous similar message [ 3508.704544] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3512.287139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3512.477385] Lustre: server umount lustre-MDT0000 complete [ 3513.824654] LustreError: 115842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3513.839241] LustreError: 115842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3516.031586] LustreError: 116576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645340 with bad export cookie 9201303879186060214 [ 3516.037577] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3516.248030] Lustre: server umount lustre-MDT0001 complete [ 3526.970727] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3529.449670] Key type lgssc unregistered [ 3529.729679] LNet: 119374:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3529.742301] LNetError: 119374:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3529.770128] LNet: Removed LNI 192.168.203.155@tcp [ 3530.428282] Key type .llcrypt unregistered [ 3530.433168] Key type ._llcrypt unregistered [ 3556.128460] Key type ._llcrypt registered [ 3556.131474] Key type .llcrypt registered [ 3556.227981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3557.273281] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3557.305079] alg: No test for adler32 (adler32-zlib) [ 3558.466507] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 3558.852865] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3560.552663] Key type lgssc registered [ 3562.133871] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3571.221759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3571.238634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3572.835806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3577.499449] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3582.445059] LustreError: 120787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3582.473426] LustreError: 120787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3585.676597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3585.963880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3589.516575] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3597.792430] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3604.458813] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3611.832495] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3612.177652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3617.337043] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3618.256491] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3625.048557] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3636.234361] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3640.821272] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3640.832165] Lustre: Skipped 2 previous similar messages [ 3643.973833] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3644.013286] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3646.371504] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3646.384115] Lustre: Skipped 2 previous similar messages [ 3649.713619] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3649.715848] Lustre: Skipped 1 previous similar message [ 3656.372776] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3656.380903] Lustre: Skipped 2 previous similar messages [ 3665.446953] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.55@tcp 192.168.0.[2,10]@tcp [ 3670.156339] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.55/24@tcp in log lustre-MDT0000 [ 3670.164793] Lustre: Skipped 2 previous similar messages [ 3677.311926] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3677.351539] Lustre: Skipped 1 previous similar message [ 3688.940488] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3688.957672] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3688.984519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3692.020471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3692.036332] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3692.193612] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3694.795596] Lustre: server umount lustre-OST0000 complete [ 3700.196444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3700.199747] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3700.236094] Lustre: Skipped 1 previous similar message [ 3700.241146] Lustre: Skipped 1 previous similar message [ 3705.312958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3705.325462] Lustre: Skipped 2 previous similar messages [ 3706.275448] Lustre: server umount lustre-MDT0000 complete [ 3709.430544] LustreError: 121854:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3709.457252] LustreError: 121854:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3710.508048] LustreError: 120768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645535 with bad export cookie 7188629405141392854 [ 3710.517285] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3710.525433] LustreError: 120768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3711.168334] Lustre: server umount lustre-MDT0001 complete [ 3724.345485] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3726.925285] Key type lgssc unregistered [ 3727.232233] LNet: 125828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3727.243365] LNetError: 125828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3727.273990] LNet: Removed LNI 192.168.203.155@tcp [ 3728.052926] Key type .llcrypt unregistered [ 3728.057510] Key type ._llcrypt unregistered [ 3760.105760] Key type ._llcrypt registered [ 3760.108272] Key type .llcrypt registered [ 3760.196890] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3761.093552] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3761.173602] alg: No test for adler32 (adler32-zlib) [ 3762.146741] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 3762.342663] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3763.984842] Key type lgssc registered [ 3764.792728] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3771.725753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3771.744841] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3773.099919] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3773.128095] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.55/24@tcp [ 3773.144401] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3777.242740] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3782.628232] LustreError: 127387:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3782.654372] LustreError: 127387:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3785.624043] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3785.908356] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3785.938960] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.55/24@tcp [ 3785.980666] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3789.946933] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3796.549644] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3802.549899] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3809.197744] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3809.569592] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3815.185893] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3817.859616] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3822.438864] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3833.312930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3833.322161] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.329070] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3833.345854] Lustre: Skipped 1 previous similar message [ 3833.359834] Lustre: Skipped 1 previous similar message [ 3838.378335] Lustre: server umount lustre-OST0000 complete [ 3842.553465] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3842.565751] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3842.592811] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3848.673467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3848.690350] Lustre: Skipped 3 previous similar messages [ 3853.793516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3853.806241] Lustre: Skipped 1 previous similar message [ 3856.351831] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3856.528106] Lustre: server umount lustre-MDT0000 complete [ 3858.914671] LustreError: 127384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3858.924024] LustreError: 127384:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3860.750258] LustreError: 127369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645685 with bad export cookie 5243182846228004207 [ 3860.764696] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3860.778731] LustreError: 127369:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3861.042435] Lustre: server umount lustre-MDT0001 complete [ 3874.928704] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 3878.959327] Key type lgssc unregistered [ 3879.365969] LNet: 130984:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3879.374074] LNetError: 130984:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3879.415379] LNet: Removed LNI 192.168.203.155@tcp [ 3880.268679] Key type .llcrypt unregistered [ 3880.270221] Key type ._llcrypt unregistered [ 3909.214504] Key type ._llcrypt registered [ 3909.220269] Key type .llcrypt registered [ 3909.322643] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 3923.243968] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 3924.075321] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3924.156272] alg: No test for adler32 (adler32-zlib) [ 3925.237919] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 3925.477492] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 3927.111222] Key type lgssc registered [ 3928.390536] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3962.834838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3962.855838] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3964.139911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3964.193682] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3964.371191] Lustre: lustre-MDT0000: new disk, initializing [ 3964.485115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3964.506929] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3969.567778] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3979.049092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3979.131417] Lustre: 134382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3979.168878] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3979.182260] Lustre: Skipped 1 previous similar message [ 3979.237719] Lustre: lustre-MDT0001: new disk, initializing [ 3979.315965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3979.347951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3979.357749] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3983.179870] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 3992.990569] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3998.978708] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4007.115931] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4007.420632] Lustre: lustre-OST0000: new disk, initializing [ 4007.424960] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4007.431996] Lustre: 136349:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4007.536472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4013.179588] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4016.674724] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4016.680648] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4016.748202] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4022.730060] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4027.251131] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4027.463508] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4031.886978] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4032.174972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4037.089284] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4037.092101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4037.101037] LustreError: Skipped 1 previous similar message [ 4037.102740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4037.110953] Lustre: Skipped 1 previous similar message [ 4037.122688] Lustre: Skipped 1 previous similar message [ 4041.045384] Lustre: server umount lustre-OST0000 complete [ 4045.798109] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4045.816946] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4045.867793] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4047.339992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4047.342322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4052.068632] Lustre: server umount lustre-MDT0000 complete [ 4056.775127] LustreError: 134374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788645881 with bad export cookie 8206002498347105109 [ 4056.776383] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4056.805092] LustreError: 134374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4057.185607] Lustre: server umount lustre-MDT0001 complete [ 4065.713515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4066.125842] LustreError: 138569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4066.141911] LustreError: 138569:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4066.186429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4070.435550] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4071.393839] LustreError: 138570:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4075.503064] LustreError: 138569:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4080.247742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4080.745981] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4084.614369] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4092.741118] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4099.160803] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4105.063578] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4105.481796] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4112.020184] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4119.378573] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4124.083836] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4124.270726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4127.603915] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4127.769933] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4134.541713] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4134.706906] Lustre: lustre-OST0001: new disk, initializing [ 4134.714799] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4134.730603] Lustre: 141940:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4134.828793] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4140.858185] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4144.797679] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4144.815560] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4144.926176] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4150.651123] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4150.995311] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4156.025213] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4156.356583] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4163.453447] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9627197e8000.ost_server_uuid 50 [ 4165.386460] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9627197e8000.ost_server_uuid in IDLE state after 0 sec [ 4175.851586] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4175.860198] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4175.892964] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4175.896053] LustreError: Skipped 1 previous similar message [ 4175.927495] Lustre: Skipped 2 previous similar messages [ 4177.534377] Lustre: server umount lustre-OST0001 complete [ 4180.974265] LustreError: 140390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4180.990461] LustreError: 140390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4182.499396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4182.503717] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4182.510516] Lustre: Skipped 1 previous similar message [ 4182.518213] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4186.087888] LustreError: 140687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4186.098201] LustreError: 140687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4187.246679] Lustre: server umount lustre-OST0000 complete [ 4191.201444] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4191.204682] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4191.207162] Lustre: Skipped 2 previous similar messages [ 4191.210693] Lustre: Skipped 3 previous similar messages [ 4196.861238] Lustre: server umount lustre-MDT0000 complete [ 4201.263281] LustreError: 138550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646026 with bad export cookie 8206002498347105991 [ 4201.266695] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4201.284870] LustreError: 138550:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4201.556699] Lustre: server umount lustre-MDT0001 complete [ 4222.886788] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4226.464895] Key type lgssc unregistered [ 4226.802490] LNet: 144941:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4226.814390] LNetError: 144941:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4226.841168] LNet: Removed LNI 192.168.203.155@tcp [ 4228.312380] Key type .llcrypt unregistered [ 4228.317425] Key type ._llcrypt unregistered [ 4267.654818] Key type ._llcrypt registered [ 4267.656544] Key type .llcrypt registered [ 4267.730935] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4283.055934] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4284.818500] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4284.847452] alg: No test for adler32 (adler32-zlib) [ 4285.913484] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 4286.183867] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4287.887196] Key type lgssc registered [ 4289.115917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4325.845241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4325.869028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4327.089717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4327.111357] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4327.183070] Lustre: lustre-MDT0000: new disk, initializing [ 4327.240734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4327.254911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4330.854655] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4340.937501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4341.058523] Lustre: 149095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4341.109538] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4341.112936] Lustre: Skipped 1 previous similar message [ 4341.167724] Lustre: lustre-MDT0001: new disk, initializing [ 4341.236287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4341.269733] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4341.282941] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4345.255340] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4353.574570] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4358.312711] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4364.220646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4364.488418] Lustre: lustre-OST0000: new disk, initializing [ 4364.492690] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4364.499913] Lustre: 151063:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4364.582106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4370.097799] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4373.605778] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4373.617197] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4373.665467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4377.902269] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4382.321115] Lustre: Failing over lustre-OST0000 [ 4384.231248] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4384.238057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4384.238282] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4384.247952] Lustre: Skipped 1 previous similar message [ 4384.265717] Lustre: Skipped 1 previous similar message [ 4384.495608] Lustre: server umount lustre-OST0000 complete [ 4406.096047] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4406.880784] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4406.900245] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4412.891424] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4415.642929] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4421.593963] Lustre: Failing over lustre-MDT0000 [ 4421.879806] Lustre: server umount lustre-MDT0000 complete [ 4423.142803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4423.161729] LustreError: Skipped 1 previous similar message [ 4423.182527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4424.242057] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4428.245028] LustreError: 149100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4428.261517] LustreError: 149100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4429.299798] LustreError: 149104:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4429.314076] LustreError: 149104:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4433.372346] LustreError: 149100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4438.487247] LustreError: 150027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4438.527894] LustreError: 150027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4440.543225] Lustre: 147316:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788646249/real 1788646249] req@00000000c7610bc9 x1875531383484544/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788646265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4440.570163] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4442.544657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4443.607276] LustreError: 150027:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.631140] LustreError: 150027:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 4450.792675] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x9227ae41cc034b0f [ 4450.802838] Lustre: MGC192.168.203.155@tcp: Connection restored to 0@lo (at 0@lo) [ 4451.010942] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4451.082055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4451.146640] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4453.843694] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4456.446305] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4456.481935] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4456.570404] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4456.762223] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4466.668172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4466.671586] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4466.673910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4466.677135] Lustre: Skipped 2 previous similar messages [ 4466.706841] LustreError: Skipped 1 previous similar message [ 4470.935690] Lustre: server umount lustre-OST0000 complete [ 4476.903569] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4476.908978] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4476.912741] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4476.912751] Lustre: Skipped 1 previous similar message [ 4476.945971] Lustre: Skipped 1 previous similar message [ 4482.016702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.024468] Lustre: Skipped 2 previous similar messages [ 4487.143494] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4487.154716] Lustre: Skipped 1 previous similar message [ 4489.186756] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4491.453865] Lustre: server umount lustre-MDT0000 complete [ 4492.263681] LustreError: 149099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4492.279653] LustreError: 149099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4495.928231] LustreError: 151062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646320 with bad export cookie 10531577851248069391 [ 4495.941184] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4495.961718] LustreError: 151062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4496.363748] Lustre: server umount lustre-MDT0001 complete [ 4509.424414] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 4512.020448] Key type lgssc unregistered [ 4512.290273] LNet: 154720:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4512.306721] LNetError: 154720:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4512.336245] LNet: Removed LNI 192.168.203.155@tcp [ 4513.206730] Key type .llcrypt unregistered [ 4513.208890] Key type ._llcrypt unregistered [ 4556.874285] Key type ._llcrypt registered [ 4556.876285] Key type .llcrypt registered [ 4556.959821] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4557.825646] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4557.832342] alg: No test for adler32 (adler32-zlib) [ 4558.808496] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 4558.966323] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 4560.615663] Key type lgssc registered [ 4561.786685] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4569.686649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4569.697980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4571.184433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4575.207168] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4580.331739] LustreError: 156247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4580.371387] LustreError: 156247:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4585.442761] LustreError: 156248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4585.996601] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4590.562817] LustreError: 156247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4590.716567] LustreError: 156276:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4590.727146] LustreError: 156276:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4590.733324] LustreError: 156276:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4591.269537] Lustre: server umount lustre-MDT0000 complete [ 4608.369243] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 4624.395993] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 4665.437937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4665.773287] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4665.809210] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4665.912299] Lustre: lustre-MDT0000: new disk, initializing [ 4665.947667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4665.975976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4670.617920] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4681.639924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4681.713595] Lustre: 159602:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4681.777425] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4681.783306] Lustre: Skipped 1 previous similar message [ 4681.896213] Lustre: lustre-MDT0001: new disk, initializing [ 4682.072612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4682.105696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4682.119867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4686.609269] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4697.128520] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4703.204818] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4710.119497] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4710.350119] Lustre: lustre-OST0000: new disk, initializing [ 4710.354186] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4710.361129] Lustre: 161572:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4710.453899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4716.477678] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 4717.605983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4717.616902] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4717.721043] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4725.521414] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4730.053554] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4731.687492] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 18:15:54 (1788646554) [ 4974.048435] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4974.048914] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4974.062904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4974.081241] Lustre: Skipped 2 previous similar messages [ 4979.168815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4979.177435] Lustre: Skipped 2 previous similar messages [ 4984.290916] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4984.303182] Lustre: Skipped 2 previous similar messages [ 4987.360746] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4987.535967] Lustre: server umount lustre-MDT0000 complete [ 4989.408839] LustreError: 159611:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4989.427293] LustreError: 159611:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4990.522605] LustreError: 159596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646815 with bad export cookie 12912090279135526164 [ 4990.524619] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4990.536356] LustreError: 159596:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4994.528254] LustreError: 159609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4994.528739] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4994.561944] Lustre: Skipped 2 previous similar messages [ 4994.569331] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4996.887388] Lustre: server umount lustre-MDT0001 complete [ 5006.012055] Lustre: server umount lustre-OST0000 complete [ 5013.731666] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5022.304931] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5057.565937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5057.877266] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5057.922786] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5058.021924] Lustre: lustre-MDT0000: new disk, initializing [ 5058.104520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5058.130758] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5063.260511] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5073.471911] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5073.597057] Lustre: 166231:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5073.652188] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5073.657477] Lustre: Skipped 1 previous similar message [ 5073.722735] Lustre: lustre-MDT0001: new disk, initializing [ 5073.797044] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5073.831966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5073.843698] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5077.698745] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5085.798987] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5090.509355] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5096.057414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5096.269763] Lustre: lustre-OST0000: new disk, initializing [ 5096.278842] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5096.294624] Lustre: 168199:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5096.398857] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5101.315835] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5106.247835] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5106.258397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5106.408361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5109.569596] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5113.288360] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5113.506227] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5117.135719] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5117.280992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5122.031097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5122.031799] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5122.038598] LustreError: Skipped 1 previous similar message [ 5122.057298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5122.075240] Lustre: Skipped 2 previous similar messages [ 5132.269137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5132.276908] Lustre: Skipped 3 previous similar messages [ 5133.279241] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5133.415306] Lustre: server umount lustre-OST0000 complete [ 5137.378410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5137.393203] Lustre: Skipped 2 previous similar messages [ 5142.416040] Lustre: server umount lustre-MDT0000 complete [ 5142.496279] LustreError: 166242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5142.512150] LustreError: 166242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5146.077245] LustreError: 166223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788646970 with bad export cookie 12912090279135684994 [ 5146.079821] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5146.087328] LustreError: 166223:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5146.489906] Lustre: server umount lustre-MDT0001 complete [ 5153.907420] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:22:56 (1788646976) [ 5167.268776] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5174.683926] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5206.892376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5207.147615] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5207.200472] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5207.293846] Lustre: lustre-MDT0000: new disk, initializing [ 5207.411537] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5207.456805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5213.569905] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5224.782313] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5224.860368] Lustre: 172958:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5225.119524] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5229.338964] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5237.773406] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5242.955173] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5248.597039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5248.814501] Lustre: lustre-OST0000: new disk, initializing [ 5248.817574] Lustre: Skipped 1 previous similar message [ 5248.819975] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5248.824218] Lustre: Skipped 2 previous similar messages [ 5248.828722] Lustre: 174933:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5253.810214] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5257.288266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5257.295135] Lustre: Skipped 1 previous similar message [ 5257.302391] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5257.376490] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5261.771298] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5267.941653] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5267.945456] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5267.979807] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5267.992197] Lustre: Skipped 3 previous similar messages [ 5274.176239] Lustre: server umount lustre-OST0000 complete [ 5278.177562] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5278.189123] Lustre: Skipped 2 previous similar messages [ 5283.508772] Lustre: server umount lustre-MDT0000 complete [ 5286.712583] LustreError: 172949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647111 with bad export cookie 12912090279135685876 [ 5286.713205] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5286.738204] LustreError: 172949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5286.942761] Lustre: server umount lustre-MDT0001 complete [ 5294.473627] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 18:25:17 (1788647117) [ 5308.843180] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 5317.136461] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5348.309446] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5348.617151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5348.656884] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5348.765388] Lustre: lustre-MDT0000: new disk, initializing [ 5348.849836] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5348.858247] Lustre: Skipped 2 previous similar messages [ 5348.870565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5352.932552] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5363.297669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5363.436794] Lustre: 179249:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5363.722048] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5367.653353] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5376.502569] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5381.476787] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5386.729737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5386.917925] Lustre: 181232:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5393.283458] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5396.561531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5396.701647] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5403.590474] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5412.321195] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5412.324227] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5412.329328] LustreError: Skipped 2 previous similar messages [ 5412.335231] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5412.344052] Lustre: Skipped 1 previous similar message [ 5412.351436] Lustre: Skipped 7 previous similar messages [ 5417.431398] Lustre: server umount lustre-OST0000 complete [ 5426.869188] Lustre: server umount lustre-MDT0000 complete [ 5427.682275] LustreError: 179258:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5427.700261] LustreError: 179258:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5430.425610] LustreError: 180054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647255 with bad export cookie 12912090279135686961 [ 5430.428080] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5430.438418] LustreError: 180054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5430.718069] Lustre: server umount lustre-MDT0001 complete [ 5444.396771] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 5447.114751] Key type lgssc unregistered [ 5447.410713] LNet: 183087:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5447.420548] LNetError: 183087:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5447.458685] LNet: Removed LNI 192.168.203.155@tcp [ 5448.103316] Key type .llcrypt unregistered [ 5448.107862] Key type ._llcrypt unregistered [ 5475.347433] Key type ._llcrypt registered [ 5475.349716] Key type .llcrypt registered [ 5475.456966] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5476.189697] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5476.245864] alg: No test for adler32 (adler32-zlib) [ 5477.302724] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 5477.476879] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 5479.135565] Key type lgssc registered [ 5480.273870] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5487.964822] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5488.007170] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.670409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5494.253972] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5499.875558] LustreError: 184499:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5499.885637] LustreError: 184499:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5502.572503] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5502.819772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5508.423676] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5517.456498] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5523.536364] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5531.479105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5531.930778] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5535.019815] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5538.306483] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5545.268531] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5565.920454] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5565.927396] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5565.940513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5567.970349] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5567.974217] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5567.988162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5571.913809] Lustre: server umount lustre-OST0000 complete [ 5578.211512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.237902] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5578.264515] Lustre: Skipped 2 previous similar messages [ 5582.708747] Lustre: server umount lustre-MDT0000 complete [ 5583.335211] LustreError: 184495:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5583.368432] LustreError: 184495:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5585.883271] LustreError: 184481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647410 with bad export cookie 17647505125030731496 [ 5585.887518] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5585.888664] LustreError: 184481:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5586.102875] Lustre: server umount lustre-MDT0001 complete [ 5597.274684] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 5599.943547] Key type lgssc unregistered [ 5600.371159] LNet: 188054:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5600.381527] LNetError: 188054:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5600.405134] LNet: Removed LNI 192.168.203.155@tcp [ 5601.267185] Key type .llcrypt unregistered [ 5601.269427] Key type ._llcrypt unregistered [ 5626.309238] Key type ._llcrypt registered [ 5626.312698] Key type .llcrypt registered [ 5626.494184] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 5628.199117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5628.349168] alg: No test for adler32 (adler32-zlib) [ 5629.461907] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 5629.738663] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 5631.463106] Key type lgssc registered [ 5632.533312] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5642.783327] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5642.808816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5644.241627] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5648.364235] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5654.498232] LustreError: 189466:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5654.511028] LustreError: 189466:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5656.167882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5656.498653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5661.078881] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5668.465356] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5674.374830] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5681.365462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5681.809605] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5687.461743] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5690.055549] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5694.815093] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5701.630630] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 5703.649518] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5703.662354] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5703.692039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5705.702561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5705.705669] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5705.709378] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5707.177808] Lustre: server umount lustre-OST0000 complete [ 5715.639302] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff962705a69000.ost_server_uuid 50 [ 5718.199656] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5733.344433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5733.356332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5733.384395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5733.393311] Lustre: Skipped 2 previous similar messages [ 5739.747380] Lustre: server umount lustre-MDT0000 complete [ 5740.512560] LustreError: 191208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5740.550456] LustreError: 191208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5744.328024] LustreError: 189446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647569 with bad export cookie 13353185474446271836 [ 5744.337779] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5744.344675] LustreError: 189446:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5744.975075] Lustre: server umount lustre-MDT0001 complete [ 5755.997172] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 18:32:57 (1788647577) [ 5763.243415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5763.800870] LustreError: 193383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5763.905075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5767.955648] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5769.205686] LustreError: 193384:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5773.281285] LustreError: 193383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5778.512201] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5779.110583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5784.282782] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5793.679072] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5801.786768] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5808.922400] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5809.123207] LustreError: 195204:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5809.130941] LustreError: 195204:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5809.209831] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5814.273920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5815.569577] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5823.836940] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5831.048265] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5831.236768] Lustre: lustre-OST0001: new disk, initializing [ 5831.239670] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5831.243369] Lustre: 196223:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5831.324629] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5836.905384] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5836.934179] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5836.958860] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5837.027240] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5845.749804] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5849.063619] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5849.080236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5849.101980] Lustre: Skipped 1 previous similar message [ 5849.114182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5849.119254] Lustre: Skipped 2 previous similar messages [ 5854.355036] Lustre: server umount lustre-OST0000 complete [ 5856.728214] LustreError: 196231:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5861.313495] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5861.451956] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5866.544333] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5866.844489] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5881.323092] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5881.327712] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5881.339245] Lustre: Skipped 2 previous similar messages [ 5881.358193] Lustre: Skipped 5 previous similar messages [ 5887.362644] Lustre: server umount lustre-OST0001 complete [ 5891.552304] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5891.566158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5891.591074] LustreError: Skipped 1 previous similar message [ 5901.799331] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5901.806401] Lustre: Skipped 7 previous similar messages [ 5905.375202] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5905.631103] Lustre: server umount lustre-MDT0000 complete [ 5906.913638] LustreError: 194383:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5906.929932] LustreError: 194383:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 19 previous similar messages [ 5909.703738] LustreError: 193364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647734 with bad export cookie 13353185474446272907 [ 5909.707431] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5909.716537] LustreError: 193364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 5910.073047] Lustre: server umount lustre-MDT0001 complete [ 5939.755016] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 18:36:01 (1788647761) [ 5946.614410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5946.861257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5946.889566] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5947.152928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5952.897460] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5962.343624] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5962.464503] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5962.521278] Lustre: 199664:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5967.765379] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5975.683811] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5981.679705] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5987.681107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5987.814113] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5993.964459] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 5994.600780] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6001.606584] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6008.557546] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6008.664960] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6015.523252] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6016.323913] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6024.395791] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6030.316621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6030.327472] Lustre: lustre-OST0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 6030.329470] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6030.329479] Lustre: Skipped 1 previous similar message [ 6030.374629] Lustre: Skipped 1 previous similar message [ 6033.715302] Lustre: server umount lustre-OST0000 complete [ 6035.411440] LustreError: 202154:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6052.327549] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6052.332448] LustreError: Skipped 2 previous similar messages [ 6055.750485] Lustre: server umount lustre-OST0001 complete [ 6062.563387] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6062.584258] Lustre: Skipped 4 previous similar messages [ 6065.468270] Lustre: server umount lustre-MDT0000 complete [ 6069.655467] LustreError: 199656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788647894 with bad export cookie 13353185474446274314 [ 6069.661338] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6069.676690] LustreError: 199656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6070.003829] Lustre: server umount lustre-MDT0001 complete [ 6096.431907] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 18:38:38 (1788647918) [ 6110.663636] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 6124.528154] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6162.869231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6163.295079] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6163.336082] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6163.472731] Lustre: lustre-MDT0000: new disk, initializing [ 6163.623961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6163.631284] Lustre: Skipped 3 previous similar messages [ 6163.647078] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6169.439895] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6180.823759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6180.900395] Lustre: 207775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6180.959426] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6180.962044] Lustre: Skipped 1 previous similar message [ 6181.029197] Lustre: lustre-MDT0001: new disk, initializing [ 6181.201159] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6181.212707] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6185.348716] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6194.274539] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6199.377885] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6205.717552] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6205.994251] Lustre: lustre-OST0000: new disk, initializing [ 6205.998118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6206.008716] Lustre: 209744:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6211.084560] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6215.752926] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6215.778858] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6215.885868] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6219.680196] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6226.213501] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6226.506404] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6231.311263] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6231.552530] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6236.650461] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6236.653061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6236.655874] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6236.655885] LustreError: Skipped 1 previous similar message [ 6236.684768] Lustre: Skipped 1 previous similar message [ 6236.734086] Lustre: Skipped 9 previous similar messages [ 6240.057714] Lustre: server umount lustre-OST0000 complete [ 6258.143603] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6258.362479] Lustre: server umount lustre-MDT0000 complete [ 6262.159198] LustreError: 207768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648086 with bad export cookie 13353185474446276099 [ 6262.180109] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6262.509929] Lustre: server umount lustre-MDT0001 complete [ 6270.921135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6271.342047] LustreError: 211961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6271.361518] LustreError: 211961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 12 previous similar messages [ 6275.877305] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6284.606885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6289.658184] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6296.801626] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6302.205544] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6309.232487] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6309.616994] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6309.629757] Lustre: Skipped 4 previous similar messages [ 6315.268744] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6322.918927] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6328.902173] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6329.263670] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6335.197977] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6335.576539] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6341.601150] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6341.628768] LustreError: Skipped 1 previous similar message [ 6352.351147] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6352.442446] Lustre: server umount lustre-OST0000 complete [ 6359.557201] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6363.656932] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6393.019653] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6397.516401] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6397.553988] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6398.809965] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6402.668335] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6402.691701] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6407.257663] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6411.038172] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6418.400761] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6418.406719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6418.423812] Lustre: Skipped 5 previous similar messages [ 6418.433357] Lustre: Skipped 12 previous similar messages [ 6421.823801] Lustre: server umount lustre-OST0000 complete [ 6433.140611] Lustre: server umount lustre-MDT0000 complete [ 6436.570769] LustreError: 211942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648261 with bad export cookie 13353185474446276974 [ 6436.574529] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6436.575639] LustreError: 211942:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6436.825189] Lustre: server umount lustre-MDT0001 complete [ 6444.206308] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 18:44:26 (1788648266) [ 6449.601030] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6453.760345] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6460.873442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6465.347773] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6472.011650] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6476.859451] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6483.333194] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6491.176641] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6498.129691] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6502.762619] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6502.979897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6506.802750] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6507.061369] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6513.408776] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6513.594336] Lustre: lustre-OST0001: new disk, initializing [ 6513.598909] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6513.604981] Lustre: 221302:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6520.139704] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6521.930012] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6521.938447] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6522.024540] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6529.077770] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6533.554342] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6533.765817] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6537.484864] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6537.670417] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6542.823749] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6542.828020] LustreError: Skipped 5 previous similar messages [ 6545.419063] Lustre: server umount lustre-OST0001 complete [ 6547.949236] LustreError: 221768:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6547.977288] LustreError: 221768:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 6551.337427] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6551.496981] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6555.701465] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6555.936212] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6583.986243] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6584.199574] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6584.210242] Lustre: Skipped 5 previous similar messages [ 6588.882912] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6589.419330] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6589.426205] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6589.434621] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6589.441476] Lustre: Skipped 1 previous similar message [ 6594.697464] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6596.036124] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6603.555341] Lustre: server umount lustre-OST0001 complete [ 6625.109831] Lustre: server umount lustre-OST0000 complete [ 6642.143153] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6642.326802] Lustre: server umount lustre-MDT0000 complete [ 6646.442063] LustreError: 217914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648471 with bad export cookie 13353185474446277947 [ 6646.447131] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6646.457301] LustreError: 217914:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6652.743826] Lustre: server umount lustre-MDT0001 complete [ 6672.533954] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 18:48:15 (1788648495) [ 6679.216958] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6679.358573] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6679.390865] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6682.779448] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6689.288405] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6689.369214] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6689.401736] Lustre: 226545:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6693.024733] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6699.306567] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6703.620308] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6708.912415] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6709.050571] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6713.969960] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6720.050889] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6727.997772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6728.055100] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6733.356110] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6738.521626] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6739.711858] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6743.358842] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6743.505100] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6746.947857] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6747.214716] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6752.045691] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff96270780d000.ost_server_uuid 50 [ 6753.557988] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff96270780d000.ost_server_uuid in IDLE state after 0 sec [ 6754.426379] Lustre: Permanently deactivating lustre-OST0001 [ 6754.437540] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6757.412456] Lustre: Permanently reactivating lustre-OST0001 [ 6761.440980] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6761.466548] Lustre: Skipped 10 previous similar messages [ 6761.478589] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6761.481975] Lustre: Skipped 18 previous similar messages [ 6767.541118] Lustre: server umount lustre-OST0001 complete [ 6777.656740] Lustre: server umount lustre-OST0000 complete [ 6790.661666] Lustre: server umount lustre-MDT0000 complete [ 6795.521335] LustreError: 226538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648620 with bad export cookie 13353185474446279228 [ 6795.527394] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6795.548200] LustreError: 226538:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6795.887076] Lustre: server umount lustre-MDT0001 complete [ 6821.075620] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 18:50:43 (1788648643) [ 6828.579289] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6828.949562] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6828.994629] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6829.010741] Lustre: Skipped 5 previous similar messages [ 6834.599214] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6843.001454] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6843.105186] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6843.129495] Lustre: 233238:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6849.295943] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6857.873061] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6864.167562] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6870.686748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6870.845074] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6870.871270] Lustre: Permanently deactivating lustre-OST0000 [ 6870.884641] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6876.236153] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6876.251452] LustreError: 235381:0:(osp_precreate.c:974:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0001: cannot cleanup orphans: rc = -5 [ 6877.915715] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6886.221410] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6893.798195] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6893.851662] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6900.023264] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6900.329554] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 6907.356314] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6909.783612] Lustre: Permanently reactivating lustre-OST0000 [ 6909.810331] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6909.822738] Lustre: Skipped 2 previous similar messages [ 6909.925422] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 6909.990565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6916.750387] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6916.761171] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6916.789642] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6923.407160] Lustre: server umount lustre-OST0001 complete [ 6927.329675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6927.342355] LustreError: Skipped 7 previous similar messages [ 6932.387313] Lustre: server umount lustre-OST0000 complete [ 6941.944324] Lustre: server umount lustre-MDT0000 complete [ 6946.003225] LustreError: 233231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648770 with bad export cookie 13353185474446280915 [ 6946.019613] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6946.020628] LustreError: 233231:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6946.544472] Lustre: server umount lustre-MDT0001 complete [ 6961.213205] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 6963.838800] Key type lgssc unregistered [ 6964.342631] LNet: 238096:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6964.347454] LNetError: 238096:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6965.430739] LNet: Removed LNI 192.168.203.155@tcp [ 6966.518086] Key type .llcrypt unregistered [ 6966.520260] Key type ._llcrypt unregistered [ 6992.166561] Key type ._llcrypt registered [ 6992.170589] Key type .llcrypt registered [ 6992.323384] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 6993.180846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6993.188846] alg: No test for adler32 (adler32-zlib) [ 6994.351505] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 6994.627995] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 6996.343181] Key type lgssc registered [ 6997.338598] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7008.212842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7008.239476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7009.827480] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7014.371665] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7020.006925] LustreError: 239561:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7020.030171] LustreError: 239561:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7021.732442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7021.811729] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7021.835201] Lustre: 239551:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7021.841579] Lustre: Permanently deactivating lustre-MDT0001 [ 7021.849570] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7022.056886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7024.043966] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7024.059166] LustreError: 239593:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7024.089663] LustreError: 239593:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7024.114626] LustreError: 239593:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 7027.147465] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7036.143588] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7042.547971] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7049.760146] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7050.062069] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7055.143633] LustreError: 241404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7055.163980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7055.779430] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7060.461361] LustreError: 241406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7062.183598] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7065.568163] LustreError: 241404:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.984729] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7068.143879] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7073.264165] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7073.279786] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7073.492016] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7079.799374] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7081.995828] Lustre: Permanently reactivating lustre-MDT0001 [ 7082.007106] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7082.013167] Lustre: Skipped 2 previous similar messages [ 7089.180360] Lustre: lustre-MDT0001: Client 998a9da8-0b31-4080-97ad-28b2619d231f (at 192.168.203.55@tcp) reconnecting [ 7095.074802] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7096.750789] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7101.234621] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7101.419558] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7104.771192] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7104.956441] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7107.126770] Lustre: Permanently deactivating lustre-MDT0001 [ 7107.146319] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7107.151904] Lustre: Skipped 2 previous similar messages [ 7115.355602] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7123.611179] Lustre: server umount lustre-MDT0000 complete [ 7124.449130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7124.459023] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7124.465847] LustreError: 239557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7124.465857] LustreError: 239557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7124.514191] Lustre: Skipped 2 previous similar messages [ 7126.836430] LustreError: 240302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788648951 with bad export cookie 14913285892130078763 [ 7126.841123] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7126.847926] LustreError: 240302:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7127.193418] Lustre: server umount lustre-MDT0001 complete [ 7130.733725] Lustre: server umount lustre-OST0000 complete [ 7141.813309] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 18:56:04 (1788648964) [ 7150.591730] Lustre: server umount lustre-OST0001 complete [ 7156.813633] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 7168.315586] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7204.942826] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7205.232151] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7205.244415] Lustre: Skipped 2 previous similar messages [ 7205.277612] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7205.374836] Lustre: lustre-MDT0000: new disk, initializing [ 7205.473921] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7205.492490] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7211.222903] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7222.974949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7223.153578] Lustre: 247838:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7223.231378] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7223.242737] Lustre: Skipped 1 previous similar message [ 7223.383850] Lustre: lustre-MDT0001: new disk, initializing [ 7223.570967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7223.597912] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7223.618212] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7228.216523] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7236.852745] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7244.036899] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7252.984259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7253.343339] Lustre: lustre-OST0000: new disk, initializing [ 7253.350567] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7253.361400] Lustre: 249808:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7253.456933] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7260.379761] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7260.407865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7260.581388] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7261.411213] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7273.473656] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7276.190844] LustreError: 250479:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7286.113697] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7286.217630] LustreError: 250479:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7286.263472] LustreError: 247845:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7286.291374] Lustre: lustre-OST0001: new disk, initializing [ 7286.294658] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7286.299706] Lustre: 250923:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7286.393049] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7292.760397] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7296.082467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7296.096291] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7296.146264] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7296.287154] LustreError: 247845:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7301.413445] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7303.174548] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7306.722421] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7306.726524] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7306.742533] LustreError: Skipped 1 previous similar message [ 7306.753482] Lustre: Skipped 1 previous similar message [ 7306.754452] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7311.229608] Lustre: server umount lustre-OST0001 complete [ 7311.846986] LustreError: 249793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.863618] LustreError: 249793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 7332.320096] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7332.320313] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7332.333624] LustreError: Skipped 1 previous similar message [ 7332.355402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7332.355525] Lustre: Skipped 2 previous similar messages [ 7332.359637] LustreError: 249791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7332.365948] Lustre: Skipped 1 previous similar message [ 7332.400979] LustreError: 249791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7337.449713] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7337.458093] Lustre: Skipped 2 previous similar messages [ 7342.563814] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7342.569300] Lustre: Skipped 1 previous similar message [ 7343.072130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7343.229311] Lustre: server umount lustre-OST0000 complete [ 7347.680958] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7347.705807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7347.713765] Lustre: Skipped 1 previous similar message [ 7355.257499] Lustre: server umount lustre-MDT0000 complete [ 7358.892811] LustreError: 247829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649183 with bad export cookie 14913285892130081360 [ 7358.897793] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7358.906495] LustreError: 247829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7373.279183] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7373.525223] Lustre: server umount lustre-MDT0001 complete [ 7383.303027] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7386.936398] Key type lgssc unregistered [ 7387.301682] LNet: 252960:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7387.319425] LNetError: 252960:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7387.337221] LNet: Removed LNI 192.168.203.155@tcp [ 7388.547096] Key type .llcrypt unregistered [ 7388.550792] Key type ._llcrypt unregistered [ 7431.409413] Key type ._llcrypt registered [ 7431.412737] Key type .llcrypt registered [ 7431.539250] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7432.904360] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7432.927167] alg: No test for adler32 (adler32-zlib) [ 7434.075697] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 7434.308591] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7435.975161] Key type lgssc registered [ 7436.943822] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7445.229793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7445.269806] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7446.604715] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7446.629856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7446.795253] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7451.265991] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7459.696142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7459.816063] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7459.881354] Lustre: 255175:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7460.290829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7464.819382] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7473.343519] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7480.423691] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7487.582391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7487.769124] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7487.948540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7494.206918] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7495.801736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7502.021960] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7557.088350] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7557.107323] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7557.120655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7557.603507] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7557.615395] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7557.642108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7559.480787] Lustre: server umount lustre-OST0000 complete [ 7563.792837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7576.517676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7577.133362] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7578.161199] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7578.183335] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7578.194394] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7583.269769] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7590.723598] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7629.284711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7629.295687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7629.300360] Lustre: Skipped 1 previous similar message [ 7634.404534] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7634.414228] Lustre: Skipped 1 previous similar message [ 7634.620285] Lustre: server umount lustre-OST0000 complete [ 7639.520680] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7639.533453] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7639.544205] Lustre: Skipped 2 previous similar messages [ 7639.577172] Lustre: Skipped 1 previous similar message [ 7649.763095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7649.768715] Lustre: Skipped 3 previous similar messages [ 7652.319282] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7652.435100] Lustre: server umount lustre-MDT0000 complete [ 7655.734770] LustreError: 255168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649480 with bad export cookie 8346668904896797225 [ 7655.739177] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7655.750130] LustreError: 255168:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7656.056942] Lustre: server umount lustre-MDT0001 complete [ 7664.509817] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7666.881151] Key type lgssc unregistered [ 7667.131768] LNet: 260601:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7667.142041] LNetError: 260601:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7667.164384] LNet: Removed LNI 192.168.203.155@tcp [ 7667.869646] Key type .llcrypt unregistered [ 7667.872190] Key type ._llcrypt unregistered [ 7687.606349] Key type ._llcrypt registered [ 7687.608922] Key type .llcrypt registered [ 7687.686194] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7688.635245] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7688.697485] alg: No test for adler32 (adler32-zlib) [ 7689.741233] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 7689.950542] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7691.655203] Key type lgssc registered [ 7692.550748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7698.369682] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7698.383338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7699.736555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7702.848567] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7709.522928] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7709.906125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7713.777209] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7720.525553] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7726.785120] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7734.279090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7734.438156] LustreError: 263814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7734.479523] LustreError: 263814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7734.821312] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7742.616835] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7746.051605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7751.394930] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7780.835472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7780.851112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7780.866721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7781.858689] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7781.868069] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7781.871475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7781.925149] Lustre: Skipped 1 previous similar message [ 7785.913544] Lustre: server umount lustre-OST0000 complete [ 7792.098021] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7792.104745] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7792.117534] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7792.139452] Lustre: Skipped 1 previous similar message [ 7792.163578] Lustre: Skipped 1 previous similar message [ 7797.217261] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7797.236075] Lustre: Skipped 1 previous similar message [ 7802.340941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7802.353953] Lustre: Skipped 1 previous similar message [ 7804.383175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7804.579490] Lustre: server umount lustre-MDT0000 complete [ 7806.436155] LustreError: 262793:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7806.459320] LustreError: 262793:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7808.332412] LustreError: 261977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649633 with bad export cookie 13108545683322062073 [ 7808.336110] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7808.343852] LustreError: 261977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7808.639884] Lustre: server umount lustre-MDT0001 complete [ 7818.801697] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7821.969102] Key type lgssc unregistered [ 7822.305817] LNet: 266305:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7822.313309] LNetError: 266305:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7822.331869] LNet: Removed LNI 192.168.203.155@tcp [ 7823.340220] Key type .llcrypt unregistered [ 7823.342455] Key type ._llcrypt unregistered [ 7840.243261] Key type ._llcrypt registered [ 7840.246650] Key type .llcrypt registered [ 7840.314708] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7841.284516] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7841.306068] alg: No test for adler32 (adler32-zlib) [ 7842.472602] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 7842.764300] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 7844.447532] Key type lgssc registered [ 7846.095884] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7855.986184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7856.035219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7857.612701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7863.794666] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7867.876984] LustreError: 267524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7867.902188] LustreError: 267524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7872.997267] LustreError: 267525:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7876.036903] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7876.948933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7882.540529] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7893.140740] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7899.436609] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7908.851722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7909.511648] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7919.092257] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 7928.747375] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7928.957821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7939.557915] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7939.565656] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7939.567519] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7939.589397] Lustre: Skipped 1 previous similar message [ 7944.676818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7944.679357] Lustre: Skipped 2 previous similar messages [ 7945.225639] Lustre: server umount lustre-OST0000 complete [ 7949.792914] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7949.827647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7954.400929] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7954.410290] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7954.432622] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7955.903535] Lustre: server umount lustre-MDT0000 complete [ 7959.934244] LustreError: 270167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649784 with bad export cookie 14569149974088318835 [ 7959.954778] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7960.312437] Lustre: server umount lustre-MDT0001 complete [ 7970.632777] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 7973.405633] Key type lgssc unregistered [ 7973.772730] LNet: 271206:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7973.776485] LNetError: 271206:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7973.796455] LNet: Removed LNI 192.168.203.155@tcp [ 7974.848698] Key type .llcrypt unregistered [ 7974.851859] Key type ._llcrypt unregistered [ 7998.034591] Key type ._llcrypt registered [ 7998.036763] Key type .llcrypt registered [ 7998.130639] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 7999.000672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7999.156830] alg: No test for adler32 (adler32-zlib) [ 8000.159988] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 8000.443496] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8002.081091] Key type lgssc registered [ 8003.545105] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8013.379275] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8013.405614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8014.841513] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8019.068644] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8025.062062] LustreError: 272622:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8025.085474] LustreError: 272622:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8027.809816] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8028.254329] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8032.731610] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8039.508582] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8045.445759] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8051.703307] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8051.950194] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8057.762237] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8060.142119] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8064.484943] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8089.057182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8089.076472] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8089.115164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8091.106969] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8091.112944] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8091.115901] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8091.134668] Lustre: Skipped 1 previous similar message [ 8095.271386] Lustre: server umount lustre-OST0000 complete [ 8099.824516] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8099.839410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8099.856988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8105.734648] Lustre: server umount lustre-MDT0000 complete [ 8106.465695] LustreError: 272621:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8106.514055] LustreError: 272621:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8110.676785] LustreError: 272601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788649935 with bad export cookie 11029826117475060431 [ 8110.682864] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8110.692978] LustreError: 272601:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8111.101161] Lustre: server umount lustre-MDT0001 complete [ 8126.830188] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8131.266333] Key type lgssc unregistered [ 8131.589579] LNet: 276925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8131.598751] LNetError: 276925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8131.615542] LNet: Removed LNI 192.168.203.155@tcp [ 8133.217768] Key type .llcrypt unregistered [ 8133.220290] Key type ._llcrypt unregistered [ 8151.054272] Key type ._llcrypt registered [ 8151.056535] Key type .llcrypt registered [ 8151.199638] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8152.121298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8152.226897] alg: No test for adler32 (adler32-zlib) [ 8153.358575] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 8153.602410] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8155.312908] Key type lgssc registered [ 8156.476453] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8163.653716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8163.674658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8165.188386] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8169.551339] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8174.563872] LustreError: 278157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8174.588682] LustreError: 278157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8178.318578] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8178.756308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8183.366386] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8191.233763] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8197.397571] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8204.503534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8204.948216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8210.104712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8211.293887] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8219.474770] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8236.003025] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8236.013205] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8236.032784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8239.079489] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8239.087122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8239.097726] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8240.371409] Lustre: server umount lustre-OST0000 complete [ 8245.216423] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8245.223500] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8245.252253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8245.269778] Lustre: Skipped 1 previous similar message [ 8249.312676] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8249.312727] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8250.801324] Lustre: server umount lustre-MDT0000 complete [ 8254.432769] LustreError: 278160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8254.452707] LustreError: 278160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8254.623639] LustreError: 278124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650079 with bad export cookie 5956075231538632498 [ 8254.632421] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8254.635291] LustreError: 278124:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8255.027899] Lustre: server umount lustre-MDT0001 complete [ 8266.361259] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8269.179904] Key type lgssc unregistered [ 8269.471916] LNet: 281856:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8269.491344] LNetError: 281856:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8269.520771] LNet: Removed LNI 192.168.203.155@tcp [ 8270.536827] Key type .llcrypt unregistered [ 8270.539294] Key type ._llcrypt unregistered [ 8281.022688] Key type ._llcrypt registered [ 8281.024854] Key type .llcrypt registered [ 8281.124753] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8299.837782] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8316.587897] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8317.549655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8317.678847] alg: No test for adler32 (adler32-zlib) [ 8318.774460] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 8319.057800] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8320.751272] Key type lgssc registered [ 8321.968490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8361.823222] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8361.870590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8363.182914] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8363.238468] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8363.331583] Lustre: lustre-MDT0000: new disk, initializing [ 8363.414497] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8363.434414] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8368.144756] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8378.664557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8378.785158] Lustre: 285409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8378.828046] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8378.831876] Lustre: Skipped 1 previous similar message [ 8378.926364] Lustre: lustre-MDT0001: new disk, initializing [ 8379.038984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8379.088434] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8379.102221] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8383.958996] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8393.721643] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8399.840657] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8407.306837] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8407.667598] Lustre: lustre-OST0000: new disk, initializing [ 8407.672472] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8407.681346] Lustre: 287376:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8407.799702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8413.250197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8413.265264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8413.442966] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8414.901332] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8424.489865] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8429.292721] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8429.549401] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8434.046194] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8434.230874] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8439.264831] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8439.268291] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8439.288861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8442.267349] Lustre: server umount lustre-OST0000 complete [ 8449.505883] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8449.533630] Lustre: Skipped 1 previous similar message [ 8449.538353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8449.546164] Lustre: Skipped 2 previous similar messages [ 8452.384376] Lustre: server umount lustre-MDT0000 complete [ 8455.832994] LustreError: 285400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650280 with bad export cookie 2252516284490770958 [ 8455.845101] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8455.848974] LustreError: 285400:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8456.118272] Lustre: server umount lustre-MDT0001 complete [ 8464.300814] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 19:18:06 (1788650286) [ 8471.151848] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8471.608877] LustreError: 289787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8471.644166] LustreError: 289787:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8471.698070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8477.160732] LustreError: 289788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8478.417226] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8481.259871] LustreError: 289787:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8486.371093] LustreError: 289788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8488.511343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8489.138952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8493.815632] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8501.689939] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8508.223506] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8515.403989] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8515.842668] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8522.152857] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8530.202173] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8547.808565] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8547.819765] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8547.834693] Lustre: Skipped 1 previous similar message [ 8547.837609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8548.833776] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8550.472895] Lustre: server umount lustre-OST0000 complete [ 8555.493984] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8555.496605] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8555.523029] Lustre: Skipped 1 previous similar message [ 8555.527816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8555.550334] Lustre: Skipped 2 previous similar messages [ 8560.687820] Lustre: server umount lustre-MDT0000 complete [ 8564.193455] LustreError: 290803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8564.210725] LustreError: 290803:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8564.666277] LustreError: 289769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650389 with bad export cookie 2252516284490771840 [ 8564.683228] LustreError: 289769:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8564.684182] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8565.200133] Lustre: server umount lustre-MDT0001 complete [ 8576.722510] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 8579.861249] Key type lgssc unregistered [ 8580.183760] LNet: 293343:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8580.201781] LNetError: 293343:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8581.221957] LNet: Removed LNI 192.168.203.155@tcp [ 8582.027254] Key type .llcrypt unregistered [ 8582.035238] Key type ._llcrypt unregistered [ 8610.039828] Key type ._llcrypt registered [ 8610.044380] Key type .llcrypt registered [ 8610.161782] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8624.998542] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8626.207198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8626.374445] alg: No test for adler32 (adler32-zlib) [ 8627.363233] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 8627.701902] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 8629.391174] Key type lgssc registered [ 8630.736401] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8676.685714] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8676.721846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8677.995535] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8678.043753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8678.135983] Lustre: lustre-MDT0000: new disk, initializing [ 8678.255015] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8678.288465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8683.046589] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8693.052449] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8693.210481] Lustre: 297163:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8693.284196] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8693.289294] Lustre: Skipped 1 previous similar message [ 8693.369974] Lustre: lustre-MDT0001: new disk, initializing [ 8693.514996] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8693.553335] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8693.563487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8697.906707] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8707.231252] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8713.236255] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8720.204951] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8720.547643] Lustre: lustre-OST2710: new disk, initializing [ 8720.551649] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8720.562185] Lustre: 299134:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8720.664450] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8727.168478] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8729.179727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8729.201273] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8729.303572] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8737.103605] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8743.124500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8743.269074] Lustre: lustre-OST03e8: new disk, initializing [ 8743.272824] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8743.277415] Lustre: 300196:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8743.359866] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8749.762524] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8753.827675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8753.848051] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8753.918867] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8759.571641] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8768.186199] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8768.536961] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8772.887640] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8773.071084] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8776.592343] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8776.787907] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8781.082837] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8781.332795] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8788.571838] Lustre: server umount lustre-MDT0000 complete [ 8789.989490] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8792.710505] LustreError: 297156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650617 with bad export cookie 4269951923097975307 [ 8792.719477] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8792.725039] LustreError: 297156:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8793.236697] Lustre: server umount lustre-MDT0001 complete [ 8806.409585] Lustre: server umount lustre-OST2710 complete [ 8819.965602] Lustre: server umount lustre-OST03e8 complete [ 8837.293692] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 8846.638087] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 8881.609332] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8881.794761] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8881.823933] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8881.875329] Lustre: lustre-MDT0000: new disk, initializing [ 8881.939654] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8881.966725] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8885.902958] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8896.674902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8896.796907] Lustre: 305447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8896.847621] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8896.851205] Lustre: Skipped 1 previous similar message [ 8896.926299] Lustre: lustre-MDT0001: new disk, initializing [ 8897.028702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8897.061242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8897.071861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8901.601660] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8910.791181] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8916.097683] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8922.709246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8922.975741] Lustre: lustre-OST0000: new disk, initializing [ 8922.982710] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8922.995157] Lustre: 307415:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8923.109951] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8928.680578] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 8931.880283] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8931.890140] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8931.998081] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8935.996748] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8940.758941] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8940.980659] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8946.224533] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8946.497360] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8952.289182] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8952.311466] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8952.334278] Lustre: Skipped 3 previous similar messages [ 8952.350341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8952.800792] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8957.925825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8957.935148] Lustre: Skipped 1 previous similar message [ 8963.039119] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8963.055535] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8963.082397] Lustre: Skipped 1 previous similar message [ 8963.357493] Lustre: server umount lustre-OST0000 complete [ 8968.165551] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8968.175060] Lustre: Skipped 1 previous similar message [ 8968.180132] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8968.187545] Lustre: Skipped 2 previous similar messages [ 8973.282609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8973.292058] Lustre: Skipped 1 previous similar message [ 8973.394904] Lustre: server umount lustre-MDT0000 complete [ 8977.138797] LustreError: 305439:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650801 with bad export cookie 4269951923097976987 [ 8977.155101] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8977.751315] Lustre: server umount lustre-MDT0001 complete [ 8987.313575] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 19:26:49 (1788650809) [ 8989.053303] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8990.489945] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 19:26:53 (1788650813) [ 8993.766948] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9021.228439] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9021.568959] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9021.613537] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9021.855750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9026.516983] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9035.438451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9035.532412] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9035.558399] Lustre: 311270:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9040.410263] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9061.739436] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9061.950799] Lustre: Denying initial registration attempt from nid 192.168.203.155@tcp, specified as failover [ 9061.956710] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9061.963879] LustreError: 312726:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -99 [ 9061.970403] LustreError: 312726:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9061.974821] LustreError: 312726:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9062.028849] Lustre: server umount lustre-OST0000 complete [ 9062.030797] LustreError: 312726:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9071.074083] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9071.076330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9071.098284] Lustre: Skipped 2 previous similar messages [ 9082.335554] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9082.503506] Lustre: server umount lustre-MDT0000 complete [ 9085.960543] LustreError: 311263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788650910 with bad export cookie 4269951923097977869 [ 9085.970537] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9085.972459] LustreError: 311263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9086.227855] Lustre: server umount lustre-MDT0001 complete [ 9097.168055] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9099.682249] Key type lgssc unregistered [ 9099.915710] LNet: 313848:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9099.924967] LNetError: 313848:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9100.970209] LNet: Removed LNI 192.168.203.155@tcp [ 9101.829663] Key type .llcrypt unregistered [ 9101.831466] Key type ._llcrypt unregistered [ 9112.838903] Key type ._llcrypt registered [ 9112.842920] Key type .llcrypt registered [ 9112.962635] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9113.939958] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9113.975664] alg: No test for adler32 (adler32-zlib) [ 9115.084803] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 9115.289034] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9116.975427] Key type lgssc registered [ 9118.157669] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9165.670530] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9173.388296] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9173.401899] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9174.658816] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9174.678877] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9174.843664] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9178.664761] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9187.468465] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9187.636273] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9187.677532] Lustre: 316612:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9187.960943] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9192.370138] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9199.471500] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9205.747903] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9225.067472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9225.333343] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9225.597513] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9231.328947] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9238.066651] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9244.642579] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9244.653688] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9244.672239] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9245.154385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9245.175839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9245.179333] Lustre: Skipped 1 previous similar message [ 9246.316794] Lustre: server umount lustre-OST0000 complete [ 9250.278558] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9250.280613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9250.304894] Lustre: Skipped 1 previous similar message [ 9255.397219] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9255.405954] Lustre: Skipped 2 previous similar messages [ 9256.008477] Lustre: server umount lustre-MDT0000 complete [ 9259.570869] LustreError: 316603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651084 with bad export cookie 9648220058332303383 [ 9259.577263] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9259.587785] LustreError: 316603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9259.943096] Lustre: server umount lustre-MDT0001 complete [ 9268.672387] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9270.866303] Key type lgssc unregistered [ 9271.049617] LNet: 320182:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9271.057376] LNetError: 320182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9271.066971] LNet: Removed LNI 192.168.203.155@tcp [ 9271.718215] Key type .llcrypt unregistered [ 9271.720629] Key type ._llcrypt unregistered [ 9294.479117] Key type ._llcrypt registered [ 9294.482660] Key type .llcrypt registered [ 9294.615950] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9295.520603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9295.539689] alg: No test for adler32 (adler32-zlib) [ 9296.612207] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 9296.962823] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9298.735592] Key type lgssc registered [ 9300.192205] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9308.105527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9308.122794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9309.360616] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9313.028111] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9319.825937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9319.904106] LustreError: 321596:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9319.938271] LustreError: 321596:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9320.110793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9323.771988] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9330.923485] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9336.835461] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9345.457263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9345.808924] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9351.842449] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9358.979380] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9374.696583] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9374.699961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9374.711570] Lustre: Skipped 2 previous similar messages [ 9374.724291] Lustre: Skipped 2 previous similar messages [ 9376.746736] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [ 9378.943279] Lustre: server umount lustre-MDT0000 complete [ 9379.817325] LustreError: 321590:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9379.844937] LustreError: 321590:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9381.853995] LustreError: 324087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9382.415783] LustreError: 322330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651207 with bad export cookie 492141714124456377 [ 9382.422097] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9382.440546] LustreError: 322330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9384.935688] LustreError: 321591:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9384.940901] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9384.989028] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9386.990659] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [ 9388.805996] Lustre: server umount lustre-MDT0001 complete [ 9394.518626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9402.444721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9407.459414] LustreError: 320905:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000095a088ab x1875536637894528/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9407.693175] LustreError: 323427:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9407.717829] LustreError: 323427:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9407.762191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9407.815956] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9411.897138] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9418.021202] LustreError: 323428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9418.028219] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9418.052490] LustreError: 323428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9420.748708] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9421.136146] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9425.604387] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9426.420741] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9433.194657] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9434.849767] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9440.000080] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9441.167548] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9446.890108] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9446.893877] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9446.897939] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9446.897946] Lustre: Skipped 1 previous similar message [ 9446.916242] Lustre: Skipped 1 previous similar message [ 9451.839548] Lustre: server umount lustre-OST0000 complete [ 9457.120693] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9457.121472] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9457.127726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9457.127735] Lustre: Skipped 1 previous similar message [ 9457.155194] Lustre: Skipped 1 previous similar message [ 9461.514483] Lustre: server umount lustre-MDT0000 complete [ 9462.268678] LustreError: 325213:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9462.287203] LustreError: 325213:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9465.139990] LustreError: 321576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651289 with bad export cookie 492141714124464007 [ 9465.164710] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9465.166444] LustreError: 321576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9465.467775] Lustre: server umount lustre-MDT0001 complete [ 9477.451760] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9479.891991] Key type lgssc unregistered [ 9480.142794] LNet: 327878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9480.151839] LNetError: 327878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9480.170431] LNet: Removed LNI 192.168.203.155@tcp [ 9481.009814] Key type .llcrypt unregistered [ 9481.011990] Key type ._llcrypt unregistered [ 9505.859934] Key type ._llcrypt registered [ 9505.861810] Key type .llcrypt registered [ 9505.982981] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9507.424578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9507.503343] alg: No test for adler32 (adler32-zlib) [ 9508.866687] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 9509.109185] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9510.863179] Key type lgssc registered [ 9511.941031] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9521.316345] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9521.327760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9522.834578] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9527.480697] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9532.395852] LustreError: 329292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9532.421187] LustreError: 329292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9536.121880] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9536.585096] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9541.229441] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9548.989158] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9554.708803] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9557.984452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9557.987516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9557.993807] Lustre: Skipped 1 previous similar message [ 9558.004244] Lustre: Skipped 1 previous similar message [ 9562.084603] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9562.088578] Lustre: Skipped 1 previous similar message [ 9563.440734] Lustre: server umount lustre-MDT0000 complete [ 9566.928589] LustreError: 329271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651391 with bad export cookie 18161872574288951110 [ 9566.929915] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9566.944250] LustreError: 329271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9567.328128] Lustre: server umount lustre-MDT0001 complete [ 9583.020610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9583.511957] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9583.584840] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9583.843123] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9589.038721] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9596.942479] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9597.033855] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9597.059750] Lustre: 331881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9597.254812] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9601.233283] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9608.620817] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9614.120368] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9623.275812] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9623.487308] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9623.503442] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9623.524835] LustreError: 333765:0:(tgt_mount.c:2617:server_fill_super()) Unable to start targets: -2 [ 9623.530080] LustreError: 333765:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9623.537614] LustreError: 333765:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9623.597378] Lustre: server umount lustre-OST0000 complete [ 9623.603985] LustreError: 333765:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9634.045873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9634.302529] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9634.550595] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9640.020450] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9640.757969] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9647.730483] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9655.506322] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9655.581369] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9655.645723] Lustre: lustre-OST0001: new disk, initializing [ 9655.649526] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9655.654991] Lustre: 335306:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9655.740425] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9661.272580] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9662.572046] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9662.587572] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9662.700234] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9669.428713] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9673.192494] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9673.197051] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9673.215425] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9673.245510] Lustre: Skipped 1 previous similar message [ 9678.307532] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9678.319496] Lustre: Skipped 1 previous similar message [ 9678.483395] Lustre: server umount lustre-OST0001 complete [ 9683.440695] LustreError: 335748:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9683.447253] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9683.448169] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9683.459949] LustreError: 335748:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9683.487573] Lustre: Skipped 2 previous similar messages [ 9688.548091] LustreError: 334233:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9688.571265] LustreError: 334233:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9693.668160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9693.672221] Lustre: Skipped 4 previous similar messages [ 9693.677790] LustreError: 336111:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9693.690191] LustreError: 336111:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9696.223348] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9696.356531] Lustre: server umount lustre-OST0000 complete [ 9699.808253] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9699.818818] LustreError: Skipped 1 previous similar message [ 9699.821893] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9703.907299] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9706.113056] Lustre: server umount lustre-MDT0000 complete [ 9709.025373] LustreError: 332996:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9709.053067] LustreError: 332996:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9709.552319] LustreError: 331872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651534 with bad export cookie 18161872574288951663 [ 9709.553645] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9709.569043] LustreError: 331872:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9709.950339] Lustre: server umount lustre-MDT0001 complete [ 9720.251638] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [ 9723.055237] Key type lgssc unregistered [ 9723.336435] LNet: 337263:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9723.343126] LNetError: 337263:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9723.358129] LNet: Removed LNI 192.168.203.155@tcp [ 9724.152755] Key type .llcrypt unregistered [ 9724.156396] Key type ._llcrypt unregistered [ 9791.064755] Key type ._llcrypt registered [ 9791.067462] Key type .llcrypt registered [ 9791.167276] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [ 9805.912725] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [ 9807.176193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9807.247285] alg: No test for adler32 (adler32-zlib) [ 9808.225395] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [ 9808.531436] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [ 9810.255406] Key type lgssc registered [ 9811.270716] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9845.088654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9845.132641] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9846.393715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9846.420939] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9846.479275] Lustre: lustre-MDT0000: new disk, initializing [ 9846.551504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9846.566093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9851.197534] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9861.909204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9862.055374] Lustre: 342193:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9862.154982] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9862.160055] Lustre: Skipped 1 previous similar message [ 9862.254209] Lustre: lustre-MDT0001: new disk, initializing [ 9862.358845] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9862.415042] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9862.435939] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9866.910161] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9876.722683] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9883.447304] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9890.850626] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9891.171402] Lustre: lustre-OST0000: new disk, initializing [ 9891.174750] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9891.181893] Lustre: 344165:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9891.299414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9898.498792] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9899.599162] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9899.622852] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9899.780506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9908.561352] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9913.734353] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9913.890395] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9919.308456] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9919.482624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9925.610311] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9925.613259] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9925.629537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9925.644615] Lustre: Skipped 1 previous similar message [ 9930.722581] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9930.735098] Lustre: Skipped 1 previous similar message [ 9935.842877] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9935.853522] Lustre: Skipped 1 previous similar message [ 9936.351136] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9936.568809] Lustre: server umount lustre-OST0000 complete [ 9940.962416] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9940.982412] Lustre: Skipped 1 previous similar message [ 9940.989919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9946.082967] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9946.098517] Lustre: Skipped 2 previous similar messages [ 9946.885885] Lustre: server umount lustre-MDT0000 complete [ 9950.906184] LustreError: 342187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651775 with bad export cookie 4510408312855883414 [ 9950.911277] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9950.919777] LustreError: 342187:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9951.258247] Lustre: server umount lustre-MDT0001 complete [ 9960.532443] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 19:43:02 (1788651782) [ 9967.976658] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 19:43:10 (1788651790) [ 9978.807898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9979.703571] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9984.992076] LustreError: 346925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9985.019378] LustreError: 346925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9985.515299] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [ 9990.130079] LustreError: 346924:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9994.872599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9995.233490] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 9995.421125] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9999.996054] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10007.478352] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10013.862967] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10020.785264] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10021.095411] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10027.682510] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10036.124445] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10047.358611] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10049.354687] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10051.133891] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10053.176395] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10055.161360] Lustre: Failing over lustre-MDT0000 [10055.416448] Lustre: server umount lustre-MDT0000 complete [10056.176703] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10056.202361] Lustre: Skipped 3 previous similar messages [10056.207825] LustreError: 346920:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10056.238321] LustreError: 346920:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10058.235615] LustreError: 348705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10063.316830] LustreError: 346919:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10063.331477] LustreError: 346919:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10071.520042] Lustre: 340417:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788651880/real 1788651880] req@00000000c82002d3 x1875537174797568/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788651896 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10071.521916] LustreError: 346925:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10071.553314] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10071.583790] LustreError: 346925:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [10074.983698] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10081.763820] LustreError: 340415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006bcaf5a7 x1875537174802560/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10082.089293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10082.149925] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10083.801133] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10087.271538] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10087.415512] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10087.452234] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10087.514696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10097.029833] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10098.481727] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10102.255891] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10104.487631] Lustre: Failing over lustre-MDT0000 [10104.712305] Lustre: server umount lustre-MDT0000 complete [10107.872618] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10107.875968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10107.893098] LustreError: 346921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10107.897661] LustreError: Skipped 1 previous similar message [10107.908200] LustreError: 346921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [10123.074679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10123.215132] Lustre: 340417:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788651932/real 1788651932] req@0000000073d4cd62 x1875537174818816/t0(0) o400->MGC192.168.203.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788651948 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10123.241277] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10123.263025] LustreError: 340415:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000472e72ea x1875537174822400/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10123.282260] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3e9830769fbe792b [10123.291108] Lustre: MGC192.168.203.155@tcp: Connection restored to 0@lo (at 0@lo) [10123.293514] Lustre: Skipped 2 previous similar messages [10123.507892] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10123.568410] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10124.764198] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10127.897506] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10128.896580] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10128.959143] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10129.042079] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10136.987252] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10138.426420] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10144.231339] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10144.232106] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10144.236050] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10144.244494] Lustre: Skipped 3 previous similar messages [10150.376532] Lustre: server umount lustre-MDT0000 complete [10153.928959] LustreError: 346906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788651978 with bad export cookie 4510408312855886123 [10153.941279] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10153.943869] LustreError: 346906:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10154.163610] Lustre: server umount lustre-MDT0001 complete [10168.644074] Lustre: server umount lustre-OST0000 complete [10188.358591] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10188.624258] LustreError: 353278:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10188.637704] LustreError: 353278:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [10188.666863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10193.770949] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10201.851124] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10207.775610] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10216.424353] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10223.532365] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10231.828445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10233.516478] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10240.196081] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10249.051953] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10254.305859] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10254.315121] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10254.336839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10254.342094] Lustre: Skipped 5 previous similar messages [10258.404571] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10260.295919] Lustre: server umount lustre-OST0000 complete [10278.367955] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10278.745960] Lustre: server umount lustre-MDT0000 complete [10278.880403] LustreError: 354520:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10278.924807] LustreError: 354520:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10282.678601] LustreError: 354014:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652107 with bad export cookie 4510408312855886872 [10282.694854] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10283.204552] Lustre: server umount lustre-MDT0001 complete [10296.161802] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10299.198571] Key type lgssc unregistered [10299.602241] LNet: 356813:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10299.609799] LNetError: 356813:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10299.631058] LNet: Removed LNI 192.168.203.155@tcp [10300.405129] Key type .llcrypt unregistered [10300.408280] Key type ._llcrypt unregistered [10330.356318] Key type ._llcrypt registered [10330.359219] Key type .llcrypt registered [10330.503375] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10331.612316] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10331.644209] alg: No test for adler32 (adler32-zlib) [10332.995458] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [10333.361885] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10335.079134] Key type lgssc registered [10336.380211] Lustre: Echo OBD driver; http://www.lustre.org/ [10344.408203] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10344.425425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10345.939669] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10350.321410] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10356.193388] LustreError: 358327:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10356.218526] LustreError: 358327:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10358.773417] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10359.256446] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10363.863134] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10371.077660] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10376.810395] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10384.175414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10384.608399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10390.691782] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10391.810456] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10397.835957] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10414.563343] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10414.569066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10414.580765] Lustre: Skipped 1 previous similar message [10414.602253] Lustre: Skipped 2 previous similar messages [10418.109293] Lustre: server umount lustre-MDT0000 complete [10419.685032] LustreError: 358328:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10419.697828] LustreError: 358328:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10421.445651] LustreError: 358308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652246 with bad export cookie 8600008824732826836 [10421.450437] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10421.454496] LustreError: 358308:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10421.816226] Lustre: server umount lustre-MDT0001 complete [10436.027601] Lustre: server umount lustre-OST0000 complete [10449.778654] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10450.316698] LustreError: 362086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10450.410394] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10455.060730] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10455.534747] LustreError: 362087:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10460.642564] LustreError: 362086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10463.164271] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10463.604937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10467.805377] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10475.150349] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10480.724810] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10486.984652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10487.449245] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10494.509908] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10495.677557] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10503.170622] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10514.913442] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10514.921307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10514.933682] Lustre: Skipped 1 previous similar message [10514.937653] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10516.448300] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10516.457285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10521.166552] Lustre: server umount lustre-MDT0000 complete [10521.572051] LustreError: 362086:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10521.595302] LustreError: 362086:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10524.474894] LustreError: 363930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652349 with bad export cookie 8600008824732827942 [10524.479769] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10524.491509] LustreError: 363930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10524.752479] Lustre: server umount lustre-MDT0001 complete [10538.444992] Lustre: server umount lustre-OST0000 complete [10552.945737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10553.435482] LustreError: 365997:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10553.500216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10558.039738] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10566.633657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10571.611392] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10578.276167] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10583.217945] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10589.009190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10589.274413] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10589.279224] Lustre: Skipped 1 previous similar message [10594.857315] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10598.516966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10601.686798] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10607.072032] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10607.077492] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10607.090045] Lustre: Skipped 1 previous similar message [10607.095521] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10607.100120] Lustre: Skipped 2 previous similar messages [10609.123733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10613.059926] Lustre: server umount lustre-OST0000 complete [10618.338043] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10618.345425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10618.367077] Lustre: Skipped 1 previous similar message [10618.376729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10618.383096] Lustre: Skipped 2 previous similar messages [10623.554851] Lustre: server umount lustre-MDT0000 complete [10624.480772] LustreError: 365998:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10624.503374] LustreError: 365998:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10627.675398] LustreError: 365977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652452 with bad export cookie 8600008824732828971 [10627.686240] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10627.690404] LustreError: 365977:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10627.953983] Lustre: server umount lustre-MDT0001 complete [10638.426360] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10642.003108] Key type lgssc unregistered [10642.520831] LNet: 369528:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10642.535571] LNetError: 369528:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10642.556925] LNet: Removed LNI 192.168.203.155@tcp [10643.417423] Key type .llcrypt unregistered [10643.419830] Key type ._llcrypt unregistered [10676.586173] Key type ._llcrypt registered [10676.592705] Key type .llcrypt registered [10676.730811] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10677.749955] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10677.837089] alg: No test for adler32 (adler32-zlib) [10678.958551] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [10679.171916] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10680.863248] Key type lgssc registered [10682.045316] Lustre: Echo OBD driver; http://www.lustre.org/ [10689.939307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10689.976527] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10690.002471] LustreError: 370966:0:(osd_handler.c:8546:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10690.041310] LustreError: 370966:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10690.049885] LustreError: 370966:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10690.056105] LustreError: 370966:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10690.064780] LustreError: 370966:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10696.750154] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10696.757182] LustreError: 371363:0:(osd_handler.c:8546:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10696.771344] LustreError: 371363:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10696.778628] LustreError: 371363:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10696.785560] LustreError: 371363:0:(tgt_mount.c:2585:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10696.794213] LustreError: 371363:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10718.433749] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [10721.511487] Key type lgssc unregistered [10722.008558] LNet: 372365:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10722.014581] LNetError: 372365:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10722.027711] LNet: Removed LNI 192.168.203.155@tcp [10723.011495] Key type .llcrypt unregistered [10723.014570] Key type ._llcrypt unregistered [10741.766725] Key type ._llcrypt registered [10741.770542] Key type .llcrypt registered [10741.877177] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [10758.664261] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [10759.401063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10759.431933] alg: No test for adler32 (adler32-zlib) [10760.526509] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [10760.767788] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [10762.463178] Key type lgssc registered [10763.611572] Lustre: Echo OBD driver; http://www.lustre.org/ [10799.200924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10799.241281] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10800.533205] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10800.577633] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10800.663846] Lustre: lustre-MDT0000: new disk, initializing [10800.739527] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10800.768314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10805.024712] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10816.034383] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10816.221042] Lustre: 375520:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10816.331495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10816.347631] Lustre: Skipped 1 previous similar message [10816.362512] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10816.427787] Lustre: lustre-MDT0001: new disk, initializing [10816.517201] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10816.552749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10816.561316] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10821.312888] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10831.344045] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10837.637715] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10843.956235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10844.271236] Lustre: lustre-OST0000: new disk, initializing [10844.275454] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10844.283324] Lustre: 377487:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10844.350159] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10849.850969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10849.858513] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10850.019156] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10850.515477] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10860.980232] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10865.935077] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10866.227346] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10871.421709] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10871.640949] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10875.875597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10875.881691] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10875.913942] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10880.182049] Lustre: server umount lustre-OST0000 complete [10886.115215] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10886.133896] Lustre: Skipped 2 previous similar messages [10886.134418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10886.146096] Lustre: Skipped 2 previous similar messages [10890.079655] Lustre: server umount lustre-MDT0000 complete [10893.777830] LustreError: 377486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652718 with bad export cookie 10169165868568064762 [10893.786772] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10893.800084] LustreError: 377486:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10894.256479] Lustre: server umount lustre-MDT0001 complete [10903.957339] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10905.930061] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 19:58:47 (1788652727) [10913.275754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10913.847688] LustreError: 379948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10913.874135] LustreError: 379948:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10913.943180] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10918.529264] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10919.392988] LustreError: 379949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10923.494825] LustreError: 379948:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.666757] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10928.011911] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10932.685022] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10940.228396] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10945.575704] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10952.567982] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10952.684727] LustreError: 381771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10952.901191] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10960.619251] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10968.398035] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10974.611728] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10974.808747] Lustre: lustre-OST0001: new disk, initializing [10974.811907] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10974.817117] Lustre: 382786:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10974.904544] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10981.374471] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [10983.498090] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10983.511818] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10983.584319] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10990.880856] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10997.744587] Lustre: lustre-OST0001: Not available for connect from 192.168.203.55@tcp (stopping) [10999.266602] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10999.266694] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10999.279229] LustreError: Skipped 1 previous similar message [10999.317270] Lustre: Skipped 1 previous similar message [11001.061894] Lustre: server umount lustre-OST0001 complete [11002.840057] LustreError: 381773:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11014.633812] LustreError: 381771:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11014.655046] LustreError: 381771:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11019.745145] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11019.755875] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11019.761137] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11019.774299] Lustre: Skipped 1 previous similar message [11019.780628] Lustre: Skipped 2 previous similar messages [11024.596644] Lustre: server umount lustre-OST0000 complete [11029.985992] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11030.007697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11030.014013] Lustre: Skipped 2 previous similar messages [11035.094219] Lustre: server umount lustre-MDT0000 complete [11035.112128] LustreError: 379949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11035.141835] LustreError: 379949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11039.803178] LustreError: 382785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788652864 with bad export cookie 10169165868568065637 [11039.804643] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11039.827073] LustreError: 382785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11046.394837] Lustre: server umount lustre-MDT0001 complete [11059.441224] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11062.451914] Key type lgssc unregistered [11062.818029] LNet: 384784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11062.838652] LNetError: 384784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11062.869863] LNet: Removed LNI 192.168.203.155@tcp [11063.630746] Key type .llcrypt unregistered [11063.633187] Key type ._llcrypt unregistered [11120.704740] Key type ._llcrypt registered [11120.707763] Key type .llcrypt registered [11120.836301] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11122.158272] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11122.208194] alg: No test for adler32 (adler32-zlib) [11123.253833] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [11123.606503] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11125.303239] Key type lgssc registered [11126.656366] Lustre: Echo OBD driver; http://www.lustre.org/ [11135.074825] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11135.107123] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11136.348365] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11136.370552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11136.552543] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11141.487429] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11150.223140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11150.357959] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11150.391859] Lustre: 387584:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11150.719072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11156.163990] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11163.933615] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11170.570701] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11175.399983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11175.430020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11180.515101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11180.524765] Lustre: Skipped 2 previous similar messages [11185.633513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11185.649811] Lustre: Skipped 1 previous similar message [11187.167121] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11187.433489] Lustre: server umount lustre-MDT0000 complete [11191.472532] LustreError: 387577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653016 with bad export cookie 8030965572191439770 [11191.481540] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11191.488943] LustreError: 387577:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11191.912565] Lustre: server umount lustre-MDT0001 complete [11202.617407] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:03:44 (1788653024) [11209.375396] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11209.929306] LustreError: 390098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11209.945909] LustreError: 390098:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11209.990212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11215.229938] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11215.333164] LustreError: 390099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11220.454834] LustreError: 390098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11224.230372] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11224.716205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11230.039195] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11238.735960] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11244.888189] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11252.315918] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11252.479760] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [11252.665318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11259.286630] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11267.241013] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11276.354192] Lustre: Permanently deactivating lustre-OST0000 [11276.372915] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [11286.042831] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11287.930365] LustreError: 393134:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [11287.940460] LustreError: 393134:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11287.949142] LustreError: 393134:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11291.318405] LustreError: 393183:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11291.325189] LustreError: 393183:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11299.279384] Lustre: server umount lustre-OST0000 complete [11301.773916] LustreError: 393432:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [11301.788752] LustreError: 393432:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [11305.444425] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11305.461238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11305.472842] Lustre: Skipped 2 previous similar messages [11305.489235] Lustre: Skipped 1 previous similar message [11310.562516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11310.574563] Lustre: Skipped 1 previous similar message [11313.735603] Lustre: server umount lustre-MDT0000 complete [11315.680278] LustreError: 391085:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11315.695043] LustreError: 391085:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11317.688556] LustreError: 390080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653142 with bad export cookie 8030965572191440351 [11317.709989] LustreError: 390080:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11317.715626] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11328.108696] Lustre: server umount lustre-MDT0001 complete [11337.105552] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11342.294887] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11344.812177] LustreError: 394590:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [11350.402514] Lustre: 394930:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.155@tcp [11352.508311] Lustre: server umount MGS complete [11365.845036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11366.492567] LustreError: 395656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11366.508537] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11366.517984] Lustre: Skipped 1 previous similar message [11366.561637] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11371.485405] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11377.121155] LustreError: 395656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11377.150369] LustreError: 395656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11380.395818] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11380.695963] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [11385.354378] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11393.254706] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11398.652433] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11400.273296] Lustre: Permanently reactivating lustre-OST0000 [11400.287851] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [11400.293926] Lustre: Skipped 2 previous similar messages [11430.879150] LustreError: 397503:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [11439.009465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11439.529502] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11439.537516] Lustre: Skipped 1 previous similar message [11449.343990] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11459.477590] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11466.209295] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11466.211916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11466.216149] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11466.221025] LustreError: Skipped 1 previous similar message [11466.253455] Lustre: Skipped 1 previous similar message [11467.165870] Lustre: server umount lustre-OST0000 complete [11472.872939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11472.888868] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11477.958118] Lustre: server umount lustre-MDT0000 complete [11481.578576] LustreError: 395656:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11481.607431] LustreError: 395656:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11482.439268] LustreError: 395637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653307 with bad export cookie 8030965572191441401 [11482.444442] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11482.455456] LustreError: 395637:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11482.767438] Lustre: server umount lustre-MDT0001 complete [11495.546624] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11498.162122] Key type lgssc unregistered [11498.515783] LNet: 400165:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11498.531873] LNetError: 400165:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11498.552812] LNet: Removed LNI 192.168.203.155@tcp [11499.478807] Key type .llcrypt unregistered [11499.481064] Key type ._llcrypt unregistered [11519.313587] Key type ._llcrypt registered [11519.315835] Key type .llcrypt registered [11519.427941] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [11536.458542] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11537.785211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11537.823053] alg: No test for adler32 (adler32-zlib) [11538.948679] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [11539.232604] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11540.913282] Key type lgssc registered [11542.113023] Lustre: Echo OBD driver; http://www.lustre.org/ [11578.174925] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11578.219046] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11579.526389] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11579.562264] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11579.638252] Lustre: lustre-MDT0000: new disk, initializing [11579.722213] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11579.740610] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11584.393792] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11596.089223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11596.232946] Lustre: 403316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11596.277747] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11596.281674] Lustre: Skipped 1 previous similar message [11596.335414] Lustre: lustre-MDT0001: new disk, initializing [11596.402978] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11596.430534] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11596.442087] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11601.215643] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11611.941790] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11618.382966] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11626.616949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11627.017486] Lustre: lustre-OST0000: new disk, initializing [11627.022617] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11627.027771] Lustre: 405285:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11627.124629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11634.206486] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11635.832397] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11635.842694] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11635.992072] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11643.397318] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11648.106314] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11648.368779] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11653.455471] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11653.721059] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11656.680880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11656.685050] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11656.692833] LustreError: Skipped 1 previous similar message [11656.699672] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11656.704340] Lustre: Skipped 1 previous similar message [11656.726582] Lustre: Skipped 1 previous similar message [11661.892573] Lustre: server umount lustre-OST0000 complete [11666.916739] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11666.955388] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11666.965068] Lustre: Skipped 1 previous similar message [11671.433923] Lustre: server umount lustre-MDT0000 complete [11672.032125] LustreError: 405292:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11672.049424] LustreError: 405292:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11675.533400] LustreError: 403309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653500 with bad export cookie 9365909118794125561 [11675.539924] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11675.546478] LustreError: 403309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11676.018354] Lustre: server umount lustre-MDT0001 complete [11685.505321] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11687.341700] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:11:49 (1788653509) [11694.229072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11694.603804] LustreError: 407745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11694.682813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11699.563574] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11699.683652] LustreError: 407746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11704.804318] LustreError: 407745:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11708.039277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11708.393434] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11713.873456] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11723.983318] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11731.381657] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11739.042373] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11739.357359] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11746.711153] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11754.882048] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11757.261902] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11759.963198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11766.240721] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11766.254979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11766.279389] Lustre: Skipped 1 previous similar message [11766.296479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11769.315661] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11769.325408] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11769.329595] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11769.357555] Lustre: Skipped 1 previous similar message [11774.438705] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11774.457904] Lustre: Skipped 1 previous similar message [11780.063657] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11780.559390] Lustre: server umount lustre-OST0000 complete [11789.793471] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11789.805538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11789.811398] Lustre: Skipped 3 previous similar messages [11792.582479] Lustre: server umount lustre-MDT0000 complete [11794.912714] LustreError: 407746:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11794.945654] LustreError: 407746:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11797.474814] LustreError: 407725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653622 with bad export cookie 9365909118794126443 [11797.497311] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11797.502372] LustreError: 407725:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11798.001514] Lustre: server umount lustre-MDT0001 complete [11809.604935] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11812.808805] Key type lgssc unregistered [11813.149764] LNet: 411474:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11813.160905] LNetError: 411474:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11814.189310] LNet: Removed LNI 192.168.203.155@tcp [11815.189818] Key type .llcrypt unregistered [11815.192251] Key type ._llcrypt unregistered [11841.011949] Key type ._llcrypt registered [11841.015339] Key type .llcrypt registered [11841.130125] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11841.462138] Key type .llcrypt unregistered [11841.464825] Key type ._llcrypt unregistered [11860.552472] Key type ._llcrypt registered [11860.558343] Key type .llcrypt registered [11860.736552] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [11862.300865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11862.355177] alg: No test for adler32 (adler32-zlib) [11863.532070] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [11863.820969] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [11865.511191] Key type lgssc registered [11866.789329] Lustre: Echo OBD driver; http://www.lustre.org/ [11875.484174] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11875.514587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11877.088343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11881.608854] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11886.578094] LustreError: 413679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11886.600791] LustreError: 413679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11890.741163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11891.111438] LustreError: 413680:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11891.128769] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11891.131704] LustreError: 413680:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11896.297598] LustreError: 413679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11896.318498] LustreError: 413679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11897.769773] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11901.415378] LustreError: 413675:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11901.429160] LustreError: 413675:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11901.463063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11905.419644] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11906.533356] LustreError: 413679:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11906.563935] LustreError: 413679:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11912.963878] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11913.391104] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11918.439798] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [11928.545210] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11928.552878] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11928.572183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11929.078156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11929.108161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11929.113573] Lustre: Skipped 1 previous similar message [11934.181760] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11939.302751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11939.319500] Lustre: Skipped 1 previous similar message [11942.372120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11942.596769] Lustre: server umount lustre-OST0000 complete [11949.047361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11949.061900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11949.071431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11949.077720] Lustre: Skipped 1 previous similar message [11952.857835] Lustre: server umount lustre-MDT0000 complete [11954.668131] LustreError: 415417:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11954.697937] LustreError: 415417:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11956.747141] LustreError: 413660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653781 with bad export cookie 9360651702861949261 [11956.752860] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11957.079599] Lustre: server umount lustre-MDT0001 complete [11967.823800] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [11971.156271] Key type lgssc unregistered [11971.577084] LNet: 416826:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11971.594473] LNetError: 416826:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11972.644859] LNet: Removed LNI 192.168.203.155@tcp [11973.810610] Key type .llcrypt unregistered [11973.813255] Key type ._llcrypt unregistered [12000.330134] Key type ._llcrypt registered [12000.333417] Key type .llcrypt registered [12000.456871] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12001.733403] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12001.768552] alg: No test for adler32 (adler32-zlib) [12002.925551] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [12003.181593] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12004.863171] Key type lgssc registered [12006.051806] Lustre: Echo OBD driver; http://www.lustre.org/ [12014.599073] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12014.641230] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12030.175542] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12035.295340] LustreError: 418214:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.155@tcp: failed processing log, type 4: rc = -110 [12061.985392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12069.231460] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12077.231825] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12083.813438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12084.301077] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12084.367873] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12089.266956] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12089.326292] LustreError: 418239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12089.353169] LustreError: 418239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12094.437873] LustreError: 418240:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12096.509768] LustreError: 418239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12096.526699] LustreError: 418239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12098.711173] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12099.102890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12104.052135] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12113.185398] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12119.855629] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12129.769625] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12129.770747] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12129.781698] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12129.800520] Lustre: Skipped 1 previous similar message [12129.816868] Lustre: Skipped 1 previous similar message [12134.191692] Lustre: server umount lustre-OST0000 complete [12140.004884] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12140.005301] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12140.009890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12140.029859] Lustre: Skipped 1 previous similar message [12143.010452] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12143.023612] Lustre: Skipped 2 previous similar messages [12143.687297] Lustre: server umount lustre-MDT0000 complete [12147.931627] LustreError: 419993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788653972 with bad export cookie 1118711561061230176 [12147.966111] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12147.967705] LustreError: 419993:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12148.649403] Lustre: server umount lustre-MDT0001 complete [12161.351639] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12165.213847] Key type lgssc unregistered [12165.574318] LNet: 421784:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12165.587614] LNetError: 421784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12165.615562] LNet: Removed LNI 192.168.203.155@tcp [12166.743479] Key type .llcrypt unregistered [12166.745632] Key type ._llcrypt unregistered [12192.029417] Key type ._llcrypt registered [12192.031829] Key type .llcrypt registered [12192.155599] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12193.414146] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12193.444734] alg: No test for adler32 (adler32-zlib) [12194.596441] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [12194.835787] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12196.519284] Key type lgssc registered [12197.666944] Lustre: Echo OBD driver; http://www.lustre.org/ [12205.030774] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12205.060146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12206.535414] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12210.924948] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12216.806353] LustreError: 423200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12216.827276] LustreError: 423200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12219.466930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12219.845127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12224.806302] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12232.603853] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12238.976499] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12246.314330] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12246.725876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12251.843500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [12253.701956] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12261.138274] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12269.563559] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.55@tcp (stopping) [12270.575641] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12270.589969] Lustre: Skipped 1 previous similar message [12270.595945] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12270.609811] Lustre: Skipped 2 previous similar messages [12272.733695] Lustre: server umount lustre-MDT0000 complete [12274.640475] LustreError: 423197:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12275.224835] LustreError: 423180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.55@tcp arrived at 1788654100 with bad export cookie 17505536338658165875 [12275.240128] LustreError: 423180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12275.699074] LustreError: 423195:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12275.715995] LustreError: 423195:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12277.303369] LustreError: 423182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654102 with bad export cookie 17505536338658164986 [12277.317160] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12277.317419] LustreError: 423182:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12280.800551] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12280.803951] LustreError: 423200:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12280.813112] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12280.844420] LustreError: 423200:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12280.886111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12283.654256] Lustre: server umount lustre-OST0000 complete [12287.968470] LustreError: 423928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12287.988205] LustreError: 423928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12290.652363] Lustre: server umount lustre-MDT0001 complete [12301.635178] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12304.184473] Key type lgssc unregistered [12304.551683] LNet: 426885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12304.559153] LNetError: 426885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12304.573799] LNet: Removed LNI 192.168.203.155@tcp [12305.413236] Key type .llcrypt unregistered [12305.415604] Key type ._llcrypt unregistered [12335.209327] Key type ._llcrypt registered [12335.211269] Key type .llcrypt registered [12335.330955] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12336.453816] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12336.479401] alg: No test for adler32 (adler32-zlib) [12337.527092] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [12337.802861] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12339.503453] Key type lgssc registered [12340.653680] Lustre: Echo OBD driver; http://www.lustre.org/ [12348.580252] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12348.593385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12349.926592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12354.165112] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12360.165748] LustreError: 428299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12360.188941] LustreError: 428299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12363.018569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12363.599544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12368.843658] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12376.929480] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12382.808740] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12389.242868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12389.623810] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12394.742982] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [12395.551328] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12402.857943] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12409.825048] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12409.837241] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [12409.837918] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12409.885670] Lustre: Skipped 1 previous similar message [12410.342931] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12410.346373] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12410.371723] Lustre: Skipped 1 previous similar message [12413.340243] Lustre: server umount lustre-MDT0001 complete [12414.950673] LustreError: 428296:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.203.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12420.582946] LustreError: 428299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12420.618648] LustreError: 428299:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12425.701499] LustreError: 428300:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12425.702977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12425.715552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12425.765098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12428.365350] Lustre: server umount lustre-OST0000 complete [12430.821778] LustreError: 428299:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12432.205884] Lustre: server umount lustre-MDT0000 complete [12445.696235] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12447.948395] Key type lgssc unregistered [12448.224766] LNet: 431983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12448.233298] LNetError: 431983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12448.246706] LNet: Removed LNI 192.168.203.155@tcp [12449.075637] Key type .llcrypt unregistered [12449.077834] Key type ._llcrypt unregistered [12472.012183] Key type ._llcrypt registered [12472.016539] Key type .llcrypt registered [12472.088681] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [12487.301150] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12488.412919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12488.440409] alg: No test for adler32 (adler32-zlib) [12489.470933] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [12489.728774] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12491.415488] Key type lgssc registered [12492.473449] Lustre: Echo OBD driver; http://www.lustre.org/ [12530.367446] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12545.811223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12545.892342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12547.165768] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12547.192528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12547.285723] Lustre: lustre-MDT0000: new disk, initializing [12547.429508] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12547.469556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12552.279718] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12568.799478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12569.122260] Lustre: lustre-OST0000: new disk, initializing [12569.126587] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12569.131879] Lustre: Skipped 1 previous similar message [12569.137587] Lustre: 437015:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12569.221525] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12575.897053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12575.908952] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12575.976705] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12576.025294] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12585.340365] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12601.567254] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12601.645416] Lustre: 435994:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12601.710904] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12601.765732] Lustre: lustre-MDT0001: new disk, initializing [12601.859331] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12601.894609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12601.910320] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12606.857909] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12616.291473] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12616.536297] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12621.486840] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12621.736907] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12628.464899] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [12629.479383] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12629.491050] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12629.497316] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12629.521074] Lustre: Skipped 1 previous similar message [12629.538577] Lustre: Skipped 1 previous similar message [12633.554587] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [12638.676933] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.55@tcp (stopping) [12638.685631] Lustre: Skipped 2 previous similar messages [12641.247129] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12641.507453] Lustre: server umount lustre-MDT0001 complete [12649.960943] LustreError: 436003:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12649.982860] LustreError: 436003:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12655.076313] LustreError: 436002:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12655.101091] LustreError: 436002:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12660.193783] LustreError: 436006:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12660.197464] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12660.226701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12660.240410] Lustre: Skipped 2 previous similar messages [12665.209786] Lustre: server umount lustre-OST0000 complete [12665.321635] LustreError: 438691:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12675.017824] Lustre: server umount lustre-MDT0000 complete [12687.975560] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12690.696156] Key type lgssc unregistered [12691.010443] LNet: 440762:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12691.016230] LNetError: 440762:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12691.041562] LNet: Removed LNI 192.168.203.155@tcp [12691.807456] Key type .llcrypt unregistered [12691.810751] Key type ._llcrypt unregistered [12732.334936] Key type ._llcrypt registered [12732.340400] Key type .llcrypt registered [12732.466469] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [12732.742036] Key type .llcrypt unregistered [12732.750630] Key type ._llcrypt unregistered [12749.114844] Key type ._llcrypt registered [12749.117331] Key type .llcrypt registered [12749.209296] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [12750.225895] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12750.347412] alg: No test for adler32 (adler32-zlib) [12751.488229] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [12751.764055] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [12753.447195] Key type lgssc registered [12754.707188] Lustre: Echo OBD driver; http://www.lustre.org/ [12784.326416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12784.346175] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12785.606473] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12785.673100] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12785.798068] Lustre: lustre-MDT0000: new disk, initializing [12785.930547] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12785.948422] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12790.511314] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12801.437851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12801.554837] Lustre: 444133:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12801.608545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12801.612307] Lustre: Skipped 1 previous similar message [12801.668580] Lustre: lustre-MDT0001: new disk, initializing [12801.753580] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12801.793234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12801.805872] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12806.104965] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12815.616768] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12822.642894] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12829.965634] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12830.326698] Lustre: lustre-OST0000: new disk, initializing [12830.331980] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12830.344987] Lustre: 446098:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12830.448318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12837.298320] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12837.451790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12837.466063] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12837.593615] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12848.022554] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12858.337015] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12858.339322] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12858.356099] Lustre: Skipped 1 previous similar message [12858.358322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12862.477148] Lustre: server umount lustre-MDT0000 complete [12866.151486] LustreError: 446097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654690 with bad export cookie 2385163070461131513 [12866.159331] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12866.163170] LustreError: 446097:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12866.383846] Lustre: server umount lustre-MDT0001 complete [12880.385026] Lustre: server umount lustre-OST0000 complete [12891.204625] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:31:53 (1788654713) [12899.978406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12900.466823] LustreError: 448125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12900.483365] LustreError: 448125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12900.529593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12904.772987] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12905.960962] LustreError: 448126:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12911.074295] LustreError: 448125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12914.700754] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12915.310373] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12920.672905] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12929.255934] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12935.687043] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12957.482139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12957.618113] Lustre: Found index 0 for lustre-OST0000, updating log [12957.776050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12963.729255] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [12971.967879] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12972.155539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12977.641515] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12977.647857] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12977.671405] Lustre: Skipped 1 previous similar message [12977.682145] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12977.689771] Lustre: Skipped 3 previous similar messages [12978.724108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12978.733336] Lustre: Skipped 1 previous similar message [12978.882752] Lustre: server umount lustre-OST0000 complete [12994.656636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12994.671095] Lustre: Skipped 1 previous similar message [12994.689573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12994.700283] Lustre: Skipped 1 previous similar message [13000.163909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13000.175595] Lustre: Skipped 1 previous similar message [13001.145287] Lustre: server umount lustre-MDT0000 complete [13004.260351] LustreError: 448125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13004.295524] LustreError: 448125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13004.813378] LustreError: 448859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788654829 with bad export cookie 2385163070461132899 [13004.830400] LustreError: 448859:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13004.831355] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13005.177977] Lustre: server umount lustre-MDT0001 complete [13014.444776] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:33:56 (1788654836) [13015.967562] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [13631.046883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13631.775442] LustreError: 454539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13631.847451] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13636.640635] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13637.097677] LustreError: 454540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13642.210731] LustreError: 454539:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13645.549243] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13645.840731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13650.748550] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13658.821394] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13664.899844] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13684.805236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13685.081529] Lustre: Found index 0 for lustre-OST0000, updating log [13685.420043] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13691.772680] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [13693.855276] LNet: 1 local NIs in recovery (showing 1): 192.168.203.155@tcp [13699.475082] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13704.168774] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13704.181437] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13704.190984] Lustre: Skipped 1 previous similar message [13704.195476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13707.662587] Lustre: server umount lustre-OST0000 complete [13712.354849] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13712.357452] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13712.373885] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13713.391744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13714.465292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13714.491497] Lustre: Skipped 3 previous similar messages [13718.564215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13718.576402] Lustre: Skipped 7 previous similar messages [13726.175135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13726.381151] Lustre: server umount lustre-MDT0000 complete [13726.752904] LustreError: 456395:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13726.785132] LustreError: 456395:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13730.073707] LustreError: 454520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655554 with bad export cookie 2385163070461133991 [13730.079425] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13730.099457] LustreError: 454520:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13730.593357] Lustre: server umount lustre-MDT0001 complete [13758.815321] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [13768.455606] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13806.008603] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 20:47:08 (1788655628) [13825.686699] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [13828.393550] Key type lgssc unregistered [13828.625603] LNet: 461150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13828.630280] LNetError: 461150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13948.455448] LNet: Removed LNI 192.168.203.155@tcp [13949.464656] Key type .llcrypt unregistered [13949.467667] Key type ._llcrypt unregistered [13964.896245] Key type ._llcrypt registered [13964.904720] Key type .llcrypt registered [13965.029285] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [13965.874447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13965.927526] alg: No test for adler32 (adler32-zlib) [13967.144900] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [13967.388765] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [13969.031130] Key type lgssc registered [13970.364344] Lustre: Echo OBD driver; http://www.lustre.org/ [13975.486338] LNet: Added LNI 192.168.203.155@tcp100 [8/256/0/180] [13976.460374] LNet: Added LNI 192.168.203.155@tcp101 [8/256/0/180] [13977.348397] LNet: Added LNI 192.168.203.155@tcp102 [8/256/0/180] [13978.192121] LNet: Added LNI 192.168.203.155@tcp103 [8/256/0/180] [13979.118664] LNet: Added LNI 192.168.203.155@tcp104 [8/256/0/180] [13979.998242] LNet: Added LNI 192.168.203.155@tcp105 [8/256/0/180] [13981.135388] LNet: Added LNI 192.168.203.155@tcp106 [8/256/0/180] [13982.168751] LNet: Added LNI 192.168.203.155@tcp107 [8/256/0/180] [13983.120946] LNet: Added LNI 192.168.203.155@tcp108 [8/256/0/180] [13983.996694] LNet: Added LNI 192.168.203.155@tcp109 [8/256/0/180] [13984.897704] LNet: Added LNI 192.168.203.155@tcp110 [8/256/0/180] [13985.809855] LNet: Added LNI 192.168.203.155@tcp111 [8/256/0/180] [13986.546687] LNet: Added LNI 192.168.203.155@tcp112 [8/256/0/180] [13987.476669] LNet: Added LNI 192.168.203.155@tcp113 [8/256/0/180] [13988.428350] LNet: Added LNI 192.168.203.155@tcp114 [8/256/0/180] [13989.277488] LNet: Added LNI 192.168.203.155@tcp115 [8/256/0/180] [13990.108283] LNet: Added LNI 192.168.203.155@tcp116 [8/256/0/180] [13991.008436] LNet: Added LNI 192.168.203.155@tcp117 [8/256/0/180] [13992.060932] LNet: Added LNI 192.168.203.155@tcp118 [8/256/0/180] [13993.034407] LNet: Added LNI 192.168.203.155@tcp119 [8/256/0/180] [13993.901729] LNet: Added LNI 192.168.203.155@tcp120 [8/256/0/180] [13994.858203] LNet: Added LNI 192.168.203.155@tcp121 [8/256/0/180] [13995.739327] LNet: Added LNI 192.168.203.155@tcp122 [8/256/0/180] [13996.556626] LNet: Added LNI 192.168.203.155@tcp123 [8/256/0/180] [13997.377916] LNet: Added LNI 192.168.203.155@tcp124 [8/256/0/180] [13998.280406] LNet: Added LNI 192.168.203.155@tcp125 [8/256/0/180] [13999.275542] LNet: Added LNI 192.168.203.155@tcp126 [8/256/0/180] [14000.042795] LNet: Added LNI 192.168.203.155@tcp127 [8/256/0/180] [14000.865832] LNet: Added LNI 192.168.203.155@tcp128 [8/256/0/180] [14001.655301] LNet: Added LNI 192.168.203.155@tcp129 [8/256/0/180] [14002.486950] LNet: Added LNI 192.168.203.155@tcp130 [8/256/0/180] [14003.272258] LNet: Added LNI 192.168.203.155@tcp131 [8/256/0/180] [14004.369737] LNet: Added LNI 192.168.203.155@tcp132 [8/256/0/180] [14005.232473] LNet: Added LNI 192.168.203.155@tcp133 [8/256/0/180] [14006.095589] LNet: Added LNI 192.168.203.155@tcp134 [8/256/0/180] [14007.002677] LNet: Added LNI 192.168.203.155@tcp135 [8/256/0/180] [14013.968835] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14013.998857] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14015.123590] Lustre: 464179:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [14015.183622] LNetError: 461634:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [14015.189180] LNetError: Protocol error connecting to 192.168.203.155@tcp118 at host 192.168.203.155:988: is it running a compatible version of Lustre? [14015.257172] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14015.294544] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14015.368816] Lustre: lustre-MDT0000: new disk, initializing [14015.424255] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14015.441980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14015.521137] LNet: 7 local NIs in recovery (showing 7): 192.168.203.155@tcp118, 192.168.203.155@tcp122, 192.168.203.155@tcp123, 192.168.203.155@tcp125, 192.168.203.155@tcp128, 192.168.203.155@tcp127, 192.168.203.155@tcp126 [14016.224164] LNetError: 464204:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [14016.232574] LNetError: 464204:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [14016.237125] LNetError: Protocol error connecting to 192.168.203.155@tcp120 at host 192.168.203.155:988: is it running a compatible version of Lustre? [14016.256817] LNetError: Skipped 11 previous similar messages [14018.608777] LNetError: 464202:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [14018.633119] LNetError: 464202:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [14018.650127] LNetError: Protocol error connecting to 192.168.203.155@tcp122 at host 192.168.203.155:988: is it running a compatible version of Lustre? [14018.661569] LNetError: Skipped 7 previous similar messages [14020.441669] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14022.688612] LNetError: 461637:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [14022.712855] LNetError: Protocol error connecting to 192.168.203.155@tcp122 at host 192.168.203.155:988: is it running a compatible version of Lustre? [14030.881028] LNetError: 461636:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [14030.889553] LNetError: Protocol error connecting to 192.168.203.155@tcp122 at host 192.168.203.155:988: is it running a compatible version of Lustre? [14031.224407] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14031.390308] Lustre: 464197:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14031.465495] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14031.472776] Lustre: Skipped 1 previous similar message [14031.626061] Lustre: lustre-MDT0001: new disk, initializing [14031.663259] Lustre: 464997:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [14031.671498] Lustre: 464997:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14031.770119] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14031.804031] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14031.814027] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14035.651234] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14040.707532] Lustre: 465427:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [14040.729712] Lustre: 465427:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14046.243166] LNetError: 464212:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [14046.250615] LNetError: Protocol error connecting to 192.168.203.155@tcp118 at host 192.168.203.155:988: is it running a compatible version of Lustre? [14046.262781] LNetError: Skipped 1 previous similar message [14049.661958] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14060.050456] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14063.712174] LNetError: 461635:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [14063.715855] LNetError: 461635:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [14063.718671] LNetError: Protocol error connecting to 192.168.203.155@tcp129 at host 192.168.203.155:988: is it running a compatible version of Lustre? [14063.746030] LNetError: Skipped 9 previous similar messages [14068.903854] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14069.268584] Lustre: lustre-OST0000: new disk, initializing [14069.273519] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14069.281028] Lustre: 466172:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14069.433830] Lustre: 466177:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [14069.443332] Lustre: 466177:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [14069.523392] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14078.050142] Lustre: 466522:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.155@tcp: can't add NID 192.168.203.155@tcp130: rc = -75 [14078.076462] Lustre: 466522:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [14078.099553] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14078.160552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14078.175672] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14078.344870] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14091.027430] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14096.486734] LNetError: 461635:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.203.155 [14096.498946] LNetError: 461635:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 17 previous similar messages [14096.520438] LNetError: Protocol error connecting to 192.168.203.155@tcp128 at host 192.168.203.155:988: is it running a compatible version of Lustre? [14096.544518] LNetError: Skipped 17 previous similar messages [14104.041224] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14104.052391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14104.059379] Lustre: Skipped 1 previous similar message [14104.082107] Lustre: Skipped 1 previous similar message [14106.067666] Lustre: server umount lustre-OST0000 complete [14113.770362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14113.790670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14113.811187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14116.496630] Lustre: server umount lustre-MDT0000 complete [14120.565163] LustreError: 464189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788655945 with bad export cookie 17092404374293155507 [14120.571956] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14120.584206] LustreError: 464189:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14121.023712] Lustre: server umount lustre-MDT0001 complete [14132.044457] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [14134.894985] Key type lgssc unregistered [14135.286830] LNet: 468086:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14135.293952] LNetError: 468086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14135.304609] LNet: Removed LNI 192.168.203.155@tcp [14136.354225] LNet: Removed LNI 192.168.203.155@tcp100 [14136.359291] LNetError: 468086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14136.366067] LNetError: 468086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14136.381416] LNet: Removed LNI 192.168.203.155@tcp101 [14137.440257] LNet: Removed LNI 192.168.203.155@tcp102 [14137.454255] LNetError: 468086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14137.459368] LNetError: 468086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [14137.465545] LNet: Removed LNI 192.168.203.155@tcp103 [14138.527743] LNet: Removed LNI 192.168.203.155@tcp104 [14139.551400] LNet: Removed LNI 192.168.203.155@tcp105 [14139.556625] LNetError: 468086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14139.569156] LNetError: 468086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [14139.577323] LNet: Removed LNI 192.168.203.155@tcp106 [14140.644023] LNet: Removed LNI 192.168.203.155@tcp107 [14141.663164] LNet: Removed LNI 192.168.203.155@tcp108 [14142.687169] LNet: Removed LNI 192.168.203.155@tcp109 [14143.711164] LNet: Removed LNI 192.168.203.155@tcp110 [14143.718780] LNetError: 468086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14143.734610] LNetError: 468086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [14143.751921] LNet: Removed LNI 192.168.203.155@tcp111 [14144.800231] LNet: Removed LNI 192.168.203.155@tcp112 [14145.823741] LNet: Removed LNI 192.168.203.155@tcp113 [14146.847622] LNet: Removed LNI 192.168.203.155@tcp114 [14147.874146] LNet: Removed LNI 192.168.203.155@tcp115 [14148.895211] LNet: Removed LNI 192.168.203.155@tcp116 [14149.919210] LNet: Removed LNI 192.168.203.155@tcp117 [14150.943503] LNet: Removed LNI 192.168.203.155@tcp118 [14151.967186] LNet: Removed LNI 192.168.203.155@tcp119 [14151.970163] LNetError: 468086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14151.979222] LNetError: 468086:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [14151.989866] LNet: Removed LNI 192.168.203.155@tcp120 [14153.055188] LNet: Removed LNI 192.168.203.155@tcp121 [14154.079144] LNet: Removed LNI 192.168.203.155@tcp122 [14155.103194] LNet: Removed LNI 192.168.203.155@tcp123 [14156.127159] LNet: Removed LNI 192.168.203.155@tcp124 [14157.151195] LNet: Removed LNI 192.168.203.155@tcp125 [14158.175156] LNet: Removed LNI 192.168.203.155@tcp126 [14159.200788] LNet: Removed LNI 192.168.203.155@tcp127 [14160.287232] LNet: Removed LNI 192.168.203.155@tcp128 [14161.311152] LNet: Removed LNI 192.168.203.155@tcp129 [14161.313769] LNet: Removed LNI 192.168.203.155@tcp130 [14161.317673] LNet: Removed LNI 192.168.203.155@tcp131 [14161.320361] LNet: Removed LNI 192.168.203.155@tcp132 [14161.322260] LNet: Removed LNI 192.168.203.155@tcp133 [14161.325722] LNet: Removed LNI 192.168.203.155@tcp134 [14161.335210] LNet: Removed LNI 192.168.203.155@tcp135 [14162.207575] Key type .llcrypt unregistered [14162.211787] Key type ._llcrypt unregistered [14188.047168] Key type ._llcrypt registered [14188.049310] Key type .llcrypt registered [14188.203609] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [14204.133247] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14205.132312] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14205.342979] alg: No test for adler32 (adler32-zlib) [14206.466683] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [14206.699490] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14208.383202] Key type lgssc registered [14209.478793] Lustre: Echo OBD driver; http://www.lustre.org/ [14245.072031] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 20:54:27 (1788656067) [14262.822257] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [14265.600177] Key type lgssc unregistered [14265.956617] LNet: 472170:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14265.971738] LNetError: 472170:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14265.994222] LNet: Removed LNI 192.168.203.155@tcp [14266.923606] Key type .llcrypt unregistered [14266.925361] Key type ._llcrypt unregistered [14284.214221] Key type ._llcrypt registered [14284.216235] Key type .llcrypt registered [14284.283527] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14284.866432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14284.961279] alg: No test for adler32 (adler32-zlib) [14285.984814] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [14286.256464] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14287.991238] Key type lgssc registered [14289.002640] Lustre: Echo OBD driver; http://www.lustre.org/ [14299.036949] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14299.067916] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14300.405064] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14300.445455] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14300.559875] Lustre: lustre-MDT0000: new disk, initializing [14300.668436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14300.704501] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14305.514134] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14315.759478] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14315.885508] Lustre: 473379:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14315.967578] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14315.974102] Lustre: Skipped 1 previous similar message [14316.057415] Lustre: lustre-MDT0001: new disk, initializing [14316.174454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14316.235457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14316.257879] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14320.312919] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14329.131518] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14334.656616] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14341.204187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14341.504282] Lustre: lustre-OST0000: new disk, initializing [14341.508303] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14341.515777] Lustre: 475350:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14341.609200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14347.943078] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14349.362786] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14349.373390] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14349.522665] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14357.265661] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14361.234459] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing lnet_if_list [14366.529859] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [14366.564845] LNet: Added LNI 192.168.203.155@tcp10 [8/256/0/180] [14371.727276] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [14371.758825] LNet: Added LNI 192.168.203.155@tcp11 [8/256/0/180] [14376.291276] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [14376.329836] LNet: Added LNI 192.168.203.155@tcp12 [8/256/0/180] [14381.579696] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [14381.618613] LNet: Added LNI 192.168.203.155@tcp13 [8/256/0/180] [14386.744873] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [14386.787769] LNet: Added LNI 192.168.203.155@tcp14 [8/256/0/180] [14392.034848] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [14392.063946] LNet: Added LNI 192.168.203.155@tcp15 [8/256/0/180] [14397.095850] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [14397.137810] LNet: Added LNI 192.168.203.155@tcp16 [8/256/0/180] [14401.967357] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [14401.988608] LNet: Added LNI 192.168.203.155@tcp17 [8/256/0/180] [14407.218983] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [14407.274540] LNet: Added LNI 192.168.203.155@tcp18 [8/256/0/180] [14412.380599] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [14412.416785] LNet: Added LNI 192.168.203.155@tcp19 [8/256/0/180] [14418.037728] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [14418.059957] LNet: Added LNI 192.168.203.155@tcp20 [8/256/0/180] [14424.107619] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [14424.137337] LNet: Added LNI 192.168.203.155@tcp21 [8/256/0/180] [14430.679585] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [14430.706819] LNet: Added LNI 192.168.203.155@tcp22 [8/256/0/180] [14436.764565] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [14436.789922] LNet: Added LNI 192.168.203.155@tcp23 [8/256/0/180] [14443.555808] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [14443.627990] LNet: Added LNI 192.168.203.155@tcp24 [8/256/0/180] [14449.965874] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [14450.028266] LNet: Added LNI 192.168.203.155@tcp25 [8/256/0/180] [14456.125330] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [14456.172945] LNet: Added LNI 192.168.203.155@tcp26 [8/256/0/180] [14462.793310] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [14462.856776] LNet: Added LNI 192.168.203.155@tcp27 [8/256/0/180] [14468.826574] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [14468.877288] LNet: Added LNI 192.168.203.155@tcp28 [8/256/0/180] [14474.765844] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [14474.801329] LNet: Added LNI 192.168.203.155@tcp29 [8/256/0/180] [14481.653333] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [14481.693438] LNet: Added LNI 192.168.203.155@tcp30 [8/256/0/180] [14489.360187] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [14489.393605] LNet: Added LNI 192.168.203.155@tcp31 [8/256/0/180] [14496.442627] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [14496.478534] LNet: Added LNI 192.168.203.155@tcp32 [8/256/0/180] [14503.914031] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [14503.968385] LNet: Added LNI 192.168.203.155@tcp33 [8/256/0/180] [14512.063287] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [14512.090865] LNet: Added LNI 192.168.203.155@tcp34 [8/256/0/180] [14521.755387] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [14521.855076] LNet: Added LNI 192.168.203.155@tcp35 [8/256/0/180] [14531.604810] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [14531.628671] LNet: Added LNI 192.168.203.155@tcp36 [8/256/0/180] [14538.752248] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [14538.781481] LNet: Added LNI 192.168.203.155@tcp37 [8/256/0/180] [14546.875441] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [14546.925567] LNet: Added LNI 192.168.203.155@tcp38 [8/256/0/180] [14554.445526] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [14554.482782] LNet: Added LNI 192.168.203.155@tcp39 [8/256/0/180] [14562.298658] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [14562.360667] LNet: Added LNI 192.168.203.155@tcp40 [8/256/0/180] [14571.305434] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [14571.387971] LNet: Added LNI 192.168.203.155@tcp41 [8/256/0/180] [14580.239064] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [14580.291416] LNet: Added LNI 192.168.203.155@tcp42 [8/256/0/180] [14588.853629] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [14588.886225] LNet: Added LNI 192.168.203.155@tcp43 [8/256/0/180] [14597.244289] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [14597.286231] LNet: Added LNI 192.168.203.155@tcp44 [8/256/0/180] [14605.296570] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [14605.319719] LNet: Added LNI 192.168.203.155@tcp45 [8/256/0/180] [14613.306401] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [14613.342678] LNet: Added LNI 192.168.203.155@tcp46 [8/256/0/180] [14621.652654] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [14621.705486] LNet: Added LNI 192.168.203.155@tcp47 [8/256/0/180] [14631.512869] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [14631.557542] LNet: Added LNI 192.168.203.155@tcp48 [8/256/0/180] [14643.294892] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [14643.363432] LNet: Added LNI 192.168.203.155@tcp49 [8/256/0/180] [14651.716361] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [14651.756762] LNet: Added LNI 192.168.203.155@tcp50 [8/256/0/180] [14674.400407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14674.409501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14674.427436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14675.424159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14675.428980] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14675.434962] Lustre: Skipped 1 previous similar message [14675.465523] Lustre: Skipped 1 previous similar message [14680.551824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14680.558842] Lustre: Skipped 3 previous similar messages [14685.151400] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14685.376873] Lustre: server umount lustre-MDT0000 complete [14689.288581] LustreError: 473373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656514 with bad export cookie 14934728487127138494 [14689.293953] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14689.304362] LustreError: 473373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14689.670440] Lustre: server umount lustre-MDT0001 complete [14707.178756] Lustre: 472653:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788656515/real 1788656515] req@000000003a5b796e x1875541870425984/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788656531 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14707.231320] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14709.210436] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14710.245726] Lustre: 472655:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788656519/real 1788656519] req@000000009ebfe8ca x1875541870426112/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788656535 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14711.328674] Lustre: 472655:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788656520/real 1788656520] req@000000000778a984 x1875541870426240/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788656536 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14715.362693] Lustre: 472652:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788656524/real 1788656524] req@00000000b91e6c54 x1875541870426496/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788656540 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14718.023102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14720.479356] Lustre: 472653:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788656529/real 1788656529] req@000000006711f2e1 x1875541870426624/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788656545 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14734.943684] LustreError: 472651:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ec4cc5a3 x1875541870427392/t0(0) o250->MGC192.168.203.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14735.352556] LustreError: 475344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14735.370137] LustreError: 475344:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [14735.418035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14737.453615] LustreError: 475345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14740.514463] LustreError: 475344:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14740.534925] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14740.590426] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14745.574561] LustreError: 475345:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14749.596510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14749.940959] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14753.011648] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14754.804357] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14762.767750] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14769.821964] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14791.648085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14791.651471] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14791.675301] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14797.284434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14797.288940] Lustre: Skipped 3 previous similar messages [14797.507338] Lustre: server umount lustre-OST0000 complete [14802.400207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14802.421596] Lustre: Skipped 2 previous similar messages [14807.522073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14807.527974] Lustre: Skipped 3 previous similar messages [14808.078407] Lustre: server umount lustre-MDT0000 complete [14812.245222] LustreError: 473373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656637 with bad export cookie 14934728487127141091 [14812.248787] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14812.264429] LustreError: 473373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14812.601245] Lustre: server umount lustre-MDT0001 complete [14824.198276] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [14827.423972] Key type lgssc unregistered [14827.874873] LNet: 502410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14827.885681] LNetError: 502410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14827.892456] LNet: Removed LNI 192.168.203.155@tcp [14827.896619] LNet: Removed LNI 192.168.203.155@tcp10 [14827.899473] LNet: Removed LNI 192.168.203.155@tcp11 [14827.904307] LNet: Removed LNI 192.168.203.155@tcp12 [14827.907955] LNet: Removed LNI 192.168.203.155@tcp13 [14827.911702] LNet: Removed LNI 192.168.203.155@tcp14 [14827.917519] LNet: Removed LNI 192.168.203.155@tcp15 [14827.923683] LNet: Removed LNI 192.168.203.155@tcp16 [14827.927727] LNet: Removed LNI 192.168.203.155@tcp17 [14827.930884] LNet: Removed LNI 192.168.203.155@tcp18 [14827.934401] LNet: Removed LNI 192.168.203.155@tcp19 [14827.937657] LNet: Removed LNI 192.168.203.155@tcp20 [14827.941444] LNet: Removed LNI 192.168.203.155@tcp21 [14827.944435] LNet: Removed LNI 192.168.203.155@tcp22 [14827.947280] LNet: Removed LNI 192.168.203.155@tcp23 [14827.949440] LNet: Removed LNI 192.168.203.155@tcp24 [14827.952297] LNet: Removed LNI 192.168.203.155@tcp25 [14827.955305] LNet: Removed LNI 192.168.203.155@tcp26 [14827.958835] LNet: Removed LNI 192.168.203.155@tcp27 [14827.962288] LNet: Removed LNI 192.168.203.155@tcp28 [14827.966390] LNet: Removed LNI 192.168.203.155@tcp29 [14827.969858] LNet: Removed LNI 192.168.203.155@tcp30 [14827.973799] LNet: Removed LNI 192.168.203.155@tcp31 [14827.976588] LNet: Removed LNI 192.168.203.155@tcp32 [14827.979325] LNet: Removed LNI 192.168.203.155@tcp33 [14827.983091] LNet: Removed LNI 192.168.203.155@tcp34 [14827.986366] LNet: Removed LNI 192.168.203.155@tcp35 [14827.990538] LNet: Removed LNI 192.168.203.155@tcp36 [14827.994776] LNet: Removed LNI 192.168.203.155@tcp37 [14827.998304] LNet: Removed LNI 192.168.203.155@tcp38 [14828.001325] LNet: Removed LNI 192.168.203.155@tcp39 [14828.004870] LNet: Removed LNI 192.168.203.155@tcp40 [14828.008379] LNet: Removed LNI 192.168.203.155@tcp41 [14828.011516] LNet: Removed LNI 192.168.203.155@tcp42 [14828.015247] LNet: Removed LNI 192.168.203.155@tcp43 [14828.019818] LNet: Removed LNI 192.168.203.155@tcp44 [14828.021598] LNet: Removed LNI 192.168.203.155@tcp45 [14828.024810] LNet: Removed LNI 192.168.203.155@tcp46 [14828.028685] LNet: Removed LNI 192.168.203.155@tcp47 [14828.033864] LNet: Removed LNI 192.168.203.155@tcp48 [14828.037417] LNet: Removed LNI 192.168.203.155@tcp49 [14828.047386] LNet: Removed LNI 192.168.203.155@tcp50 [14828.850931] Key type .llcrypt unregistered [14828.853447] Key type ._llcrypt unregistered [14848.427156] Key type ._llcrypt registered [14848.434547] Key type .llcrypt registered [14848.546835] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [14863.675713] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [14864.997171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14865.042228] alg: No test for adler32 (adler32-zlib) [14866.097467] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [14866.333981] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [14868.015164] Key type lgssc registered [14869.235639] Lustre: Echo OBD driver; http://www.lustre.org/ [14907.956054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14907.980753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14909.277775] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14909.303448] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14909.399378] Lustre: lustre-MDT0000: new disk, initializing [14909.464698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14909.508842] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14914.968840] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14926.218807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14926.318963] Lustre: 505609:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14926.363758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14926.366618] Lustre: Skipped 1 previous similar message [14926.414695] Lustre: lustre-MDT0001: new disk, initializing [14926.527255] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14926.616990] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14926.627545] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14933.689186] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14948.904168] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14956.118694] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14965.067078] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14965.328948] Lustre: *** cfs_fail_loc=90f, val=0*** [14965.474952] Lustre: lustre-OST0000: new disk, initializing [14965.481054] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14965.496486] Lustre: 507624:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14965.665471] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14970.978521] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14970.997951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14971.114554] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14972.808431] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [14983.014982] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14991.857220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14991.858544] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14991.869348] LustreError: Skipped 1 previous similar message [14991.871179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14991.900227] Lustre: Skipped 1 previous similar message [14996.862327] Lustre: server umount lustre-OST0000 complete [15002.084236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15002.090701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15002.100591] Lustre: Skipped 1 previous similar message [15002.129219] Lustre: Skipped 2 previous similar messages [15007.200469] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15007.204522] Lustre: Skipped 1 previous similar message [15011.300710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15011.315497] Lustre: Skipped 1 previous similar message [15015.391117] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15015.563602] Lustre: server umount lustre-MDT0000 complete [15019.700632] LustreError: 507623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788656844 with bad export cookie 1621666408758248588 [15019.705742] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15019.720091] LustreError: 507623:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15020.160960] Lustre: server umount lustre-MDT0001 complete [15032.684832] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [15035.651733] Key type lgssc unregistered [15036.013783] LNet: 509533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15036.020618] LNetError: 509533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15036.073794] LNet: Removed LNI 192.168.203.155@tcp [15037.106158] Key type .llcrypt unregistered [15037.108123] Key type ._llcrypt unregistered [15060.213513] Key type ._llcrypt registered [15060.217309] Key type .llcrypt registered [15060.376672] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [15061.756494] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15061.886885] alg: No test for adler32 (adler32-zlib) [15063.131327] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [15063.505487] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [15065.239342] Key type lgssc registered [15066.551115] Lustre: Echo OBD driver; http://www.lustre.org/ [15075.438248] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15075.462919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15077.033297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15082.071971] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15086.565575] LustreError: 510800:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15086.601340] LustreError: 510800:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [15091.687943] LustreError: 510801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15091.740599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15092.091792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15097.098330] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15105.023419] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15111.511979] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15118.723253] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15119.036335] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15125.266107] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15134.574646] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15148.417697] loop0: detected capacity change from 0 to 204800000 [15148.603506] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15148.625334] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15148.635883] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15155.687443] loop0: detected capacity change from 0 to 204800000 [15155.840650] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15155.871402] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15155.896204] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [15161.282506] loop0: detected capacity change from 0 to 204800000 [15167.182308] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15167.308893] Lustre: fs15246-OST0000: new disk, initializing [15167.314582] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [15167.322519] Lustre: 514396:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15167.360412] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15175.531888] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15187.832464] loop1: detected capacity change from 0 to 204800000 [15193.935836] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15194.005418] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [15194.104966] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [15194.286972] Lustre: fs15246-MDT0000: new disk, initializing [15194.519231] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15194.545489] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15194.673666] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [15195.745945] Lustre: 515519:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000006811e3ad x1875542685042688/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788657036 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [15198.391807] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15204.850323] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [15204.861814] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [15204.891965] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [15227.373839] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15227.405287] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [15228.836864] Lustre: server umount fs15246-MDT0000 complete [15232.509673] LustreError: 510795:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15237.614989] LustreError: 510795:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15238.918695] Lustre: server umount fs15246-OST0000 complete [15252.967781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15252.988850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15252.996154] Lustre: Skipped 1 previous similar message [15253.016971] Lustre: Skipped 1 previous similar message [15257.656713] Lustre: server umount lustre-OST0000 complete [15263.203902] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15263.210892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15263.228610] Lustre: Skipped 1 previous similar message [15263.245603] Lustre: Skipped 1 previous similar message [15267.925566] Lustre: server umount lustre-MDT0000 complete [15268.328852] LustreError: 510795:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15268.355907] LustreError: 510795:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15272.148104] LustreError: 510781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657096 with bad export cookie 12735110590014488263 [15272.151112] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15272.161824] LustreError: 510781:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15272.511694] Lustre: server umount lustre-MDT0001 complete [15285.723642] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [15289.365559] Key type lgssc unregistered [15289.794584] LNet: 518615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15289.807086] LNetError: 518615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15289.816777] LNet: Removed LNI 192.168.203.155@tcp [15291.190417] Key type .llcrypt unregistered [15291.195432] Key type ._llcrypt unregistered [15314.370420] Key type ._llcrypt registered [15314.373691] Key type .llcrypt registered [15314.577600] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [15316.485613] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15316.612780] alg: No test for adler32 (adler32-zlib) [15317.734541] Lustre: Lustre: Build Version: 2.17.58_3_ge47db3b [15317.975339] LNet: Added LNI 192.168.203.155@tcp [8/256/0/180] [15319.631229] Key type lgssc registered [15321.853933] Lustre: Echo OBD driver; http://www.lustre.org/ [15363.281032] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [15374.579977] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [15413.633976] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15413.683675] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15414.989928] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15415.059222] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15415.166811] Lustre: lustre-MDT0000: new disk, initializing [15415.264097] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15415.295111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15420.056492] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15430.456688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15430.588723] Lustre: 523081:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15430.676543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15430.687289] Lustre: Skipped 1 previous similar message [15430.786344] Lustre: lustre-MDT0001: new disk, initializing [15430.946260] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15430.997193] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15431.020803] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15435.484575] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15445.889067] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15451.880496] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15458.716387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15459.005771] Lustre: lustre-OST0000: new disk, initializing [15459.008789] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15459.017212] Lustre: 525048:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15459.113147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15465.506229] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_default_debug -1 all [15466.043292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15466.049819] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15466.176360] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15475.022982] Lustre: DEBUG MARKER: oleg355-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15479.963344] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15480.212117] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15484.407848] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15484.595182] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15486.946200] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15486.955116] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15486.958360] LustreError: Skipped 1 previous similar message [15486.978515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15486.981461] Lustre: Skipped 1 previous similar message [15486.994713] Lustre: Skipped 1 previous similar message [15492.065748] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15492.073786] Lustre: Skipped 1 previous similar message [15492.461208] Lustre: server umount lustre-OST0000 complete [15497.187393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15497.200089] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15497.213967] Lustre: Skipped 1 previous similar message [15502.304646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15502.311851] Lustre: Skipped 2 previous similar messages [15503.087308] Lustre: server umount lustre-MDT0000 complete [15506.667996] LustreError: 526588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788657331 with bad export cookie 5801729078107733072 [15506.669949] LustreError: MGC192.168.203.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15506.676928] LustreError: 526588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15507.068954] Lustre: server umount lustre-MDT0001 complete [15534.481410] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing set_hostid [15545.070621] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing load_modules_local [15580.326041] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 15329 sec ========= 21:16:42 (1788657402) [15582.605290] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:16:44 (1788657404) === [15586.468923] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:16:48 (1788657408) === [15632.843919] Lustre: DEBUG MARKER: oleg355-server.virtnet: executing unload_modules_local [15636.120553] Key type lgssc unregistered [15636.452303] LNet: 531693:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15636.461802] LNetError: 531693:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15636.482257] LNet: Removed LNI 192.168.203.155@tcp [15637.688402] Key type .llcrypt unregistered [15637.691818] Key type ._llcrypt unregistered