[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 475270210 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003307] x2apic enabled [ 0.004010] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.007877] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009014] pid_max: default: 32768 minimum: 301 [ 0.011137] LSM: Security Framework initializing [ 0.012060] Yama: becoming mindful. [ 0.013036] SELinux: Initializing. [ 0.014079] *** VALIDATE selinux *** [ 0.022503] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.027061] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028139] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029100] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030123] *** VALIDATE tmpfs *** [ 0.031488] *** VALIDATE proc *** [ 0.033273] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.035239] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036156] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037010] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038032] Spectre V2 : User space: Vulnerable [ 0.039010] Speculative Store Bypass: Vulnerable [ 0.042291] debug: unmapping init [mem 0xffffffff9a859000-0xffffffff9a860fff] [ 0.044192] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045724] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046026] ... version: 2 [ 0.047016] ... bit width: 48 [ 0.048013] ... generic registers: 4 [ 0.049014] ... value mask: 0000ffffffffffff [ 0.050017] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052014] ... event mask: 000000070000000f [ 0.054461] rcu: Hierarchical SRCU implementation. [ 0.056556] smp: Bringing up secondary CPUs ... [ 0.057684] x86: Booting SMP configuration: [ 0.058033] .... node #0, CPUs: #1 #2 #3 [ 0.062045] smp: Brought up 1 node, 4 CPUs [ 0.064024] smpboot: Max logical packages: 1 [ 0.065021] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.260032] node 0 deferred pages initialised in 193ms [ 0.263337] devtmpfs: initialized [ 0.264246] x86/mm: Memory block size: 128MB [ 0.266706] gcov: version magic: 0x41383552 [ 0.267606] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.270119] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.272526] pinctrl core: initialized pinctrl subsystem [ 0.275292] [ 0.275806] ************************************************************* [ 0.278019] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.280016] ** ** [ 0.283019] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.285016] ** ** [ 0.287021] ** This means that this kernel is built to expose internal ** [ 0.290018] ** IOMMU data structures, which may compromise security on ** [ 0.292015] ** your system. ** [ 0.294020] ** ** [ 0.297020] ** If you see this message and you are not debugging the ** [ 0.299017] ** kernel, report this immediately to your vendor! ** [ 0.300022] ** ** [ 0.303016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.305014] ************************************************************* [ 0.308000] NET: Registered protocol family 16 [ 0.310540] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.313076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.316081] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.320115] cpuidle: using governor menu [ 0.321892] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.324707] PCI: Using configuration type 1 for base access [ 0.327151] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.337062] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.338026] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.340096] cryptd: max_cpu_qlen set to 1000 [ 0.343297] ACPI: Added _OSI(Module Device) [ 0.345021] ACPI: Added _OSI(Processor Device) [ 0.346038] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.347013] ACPI: Added _OSI(Processor Aggregator Device) [ 0.352388] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.359632] ACPI: Interpreter enabled [ 0.361081] ACPI: PM: (supports S0 S3 S4 S5) [ 0.363021] ACPI: Using IOAPIC for interrupt routing [ 0.365153] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.368453] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.380664] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.383054] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.385026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.389110] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.394458] acpiphp: Slot [2] registered [ 0.395140] acpiphp: Slot [5] registered [ 0.397129] acpiphp: Slot [6] registered [ 0.398128] acpiphp: Slot [7] registered [ 0.400137] acpiphp: Slot [8] registered [ 0.401173] acpiphp: Slot [9] registered [ 0.403136] acpiphp: Slot [10] registered [ 0.404113] acpiphp: Slot [3] registered [ 0.406115] acpiphp: Slot [4] registered [ 0.407115] acpiphp: Slot [11] registered [ 0.409130] acpiphp: Slot [12] registered [ 0.410164] acpiphp: Slot [13] registered [ 0.411134] acpiphp: Slot [14] registered [ 0.413110] acpiphp: Slot [15] registered [ 0.414107] acpiphp: Slot [16] registered [ 0.416123] acpiphp: Slot [17] registered [ 0.417154] acpiphp: Slot [18] registered [ 0.418104] acpiphp: Slot [19] registered [ 0.420149] acpiphp: Slot [20] registered [ 0.421146] acpiphp: Slot [21] registered [ 0.423170] acpiphp: Slot [22] registered [ 0.424111] acpiphp: Slot [23] registered [ 0.426117] acpiphp: Slot [24] registered [ 0.427097] acpiphp: Slot [25] registered [ 0.428115] acpiphp: Slot [26] registered [ 0.429123] acpiphp: Slot [27] registered [ 0.430128] acpiphp: Slot [28] registered [ 0.432115] acpiphp: Slot [29] registered [ 0.433130] acpiphp: Slot [30] registered [ 0.435092] acpiphp: Slot [31] registered [ 0.436047] PCI host bridge to bus 0000:00 [ 0.437016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.439025] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.440023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.442029] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.444023] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.446025] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.447187] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.449814] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.452066] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.460021] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.465064] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.467022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.469019] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.471018] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.473603] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.475844] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.478055] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.481766] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.487021] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.501021] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.506019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.511967] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.532028] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.565028] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.614025] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.629000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.642027] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.653023] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.681028] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.695474] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.729032] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.743023] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.771022] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.788044] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.798023] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.807029] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.829024] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.841860] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.847023] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.856031] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.895027] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.907982] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.915022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.923033] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.942030] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.957286] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.958373] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.960462] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.962307] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.964197] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.968085] iommu: Default domain type: Passthrough [ 0.970546] SCSI subsystem initialized [ 0.971136] ACPI: bus type USB registered [ 0.972115] usbcore: registered new interface driver usbfs [ 0.974223] usbcore: registered new interface driver hub [ 0.976122] usbcore: registered new device driver usb [ 0.978144] pps_core: LinuxPPS API ver. 1 registered [ 0.980015] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.982160] PTP clock support registered [ 0.983186] EDAC MC: Ver: 3.0.0 [ 0.985600] PCI: Using ACPI for IRQ routing [ 0.987910] NetLabel: Initializing [ 0.989013] NetLabel: domain hash size = 128 [ 0.991015] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.993085] NetLabel: unlabeled traffic allowed by default [ 0.995145] vgaarb: loaded [ 0.996369] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.998017] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 1.003434] clocksource: Switched to clocksource kvm-clock [ 1.123423] VFS: Disk quotas dquot_6.6.0 [ 1.125210] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.128326] *** VALIDATE ramfs *** [ 1.129762] *** VALIDATE hugetlbfs *** [ 1.131584] pnp: PnP ACPI init [ 1.134394] pnp: PnP ACPI: found 6 devices [ 1.152866] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.155228] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.156700] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.158052] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.159403] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.161237] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.163863] NET: Registered protocol family 2 [ 1.167180] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.176400] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.181613] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.188670] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.193585] TCP: Hash tables configured (established 65536 bind 65536) [ 1.197447] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.201369] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.204521] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.208461] NET: Registered protocol family 1 [ 1.212269] RPC: Registered named UNIX socket transport module. [ 1.214670] RPC: Registered udp transport module. [ 1.216392] RPC: Registered tcp transport module. [ 1.217552] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.219095] NET: Registered protocol family 44 [ 1.220252] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.221729] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.223940] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.225981] PCI: CLS 0 bytes, default 64 [ 1.227687] Unpacking initramfs... [ 2.715924] debug: unmapping init [mem 0xffff92207cc54000-0xffff92207ffbffff] [ 2.719908] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.722259] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.725310] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.253931] Initialise system trusted keyrings [ 3.255689] Key type blacklist registered [ 3.257769] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.266674] zbud: loaded [ 3.270074] *** VALIDATE nfs *** [ 3.271362] *** VALIDATE nfs4 *** [ 3.273058] pstore: using deflate compression [ 3.276693] Platform Keyring initialized [ 3.390534] NET: Registered protocol family 38 [ 3.392519] Key type asymmetric registered [ 3.394149] Asymmetric key parser 'x509' registered [ 3.395970] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.399209] io scheduler mq-deadline registered [ 3.400947] io scheduler kyber registered [ 3.402544] io scheduler bfq registered [ 3.405183] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.408275] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.411601] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.414271] ACPI: Power Button [PWRF] [ 3.420109] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.427982] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.442173] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.455811] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.470847] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.500155] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.528975] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.533719] Non-volatile memory driver v1.3 [ 3.535193] Linux agpgart interface v0.103 [ 3.568387] virtio_blk virtio1: [vda] 146016 512-byte logical blocks (74.8 MB/71.3 MiB) [ 3.571304] vda: detected capacity change from 0 to 74760192 [ 3.586323] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.589721] vdb: detected capacity change from 0 to 1073741824 [ 3.609124] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.611911] vdc: detected capacity change from 0 to 2621440000 [ 3.625561] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.628225] vdd: detected capacity change from 0 to 2621440000 [ 3.642535] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.645318] vde: detected capacity change from 0 to 4294967296 [ 3.659863] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.662259] vdf: detected capacity change from 0 to 4294967296 [ 3.668411] libphy: Fixed MDIO Bus: probed [ 3.681874] usbcore: registered new interface driver usbserial_generic [ 3.684075] usbserial: USB Serial support registered for generic [ 3.685946] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.689128] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.690316] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.692107] mousedev: PS/2 mouse device common for all mice [ 3.694234] rtc_cmos 00:05: RTC can wake from S4 [ 3.696380] rtc_cmos 00:05: registered as rtc0 [ 3.697882] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.698028] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.699810] intel_pstate: CPU model not supported [ 3.705494] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.707744] hid: raw HID events driver (C) Jiri Kosina [ 3.710606] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.710897] usbcore: registered new interface driver usbhid [ 3.714826] usbhid: USB HID core driver [ 3.716539] drop_monitor: Initializing network drop monitor service [ 3.719066] Initializing XFRM netlink socket [ 3.720965] NET: Registered protocol family 10 [ 3.723635] Segment Routing with IPv6 [ 3.724907] NET: Registered protocol family 17 [ 3.726552] mpls_gso: MPLS GSO support [ 3.732316] RAS: Correctable Errors collector initialized. [ 3.734504] AVX version of gcm_enc/dec engaged. [ 3.736169] AES CTR mode by8 optimization enabled [ 3.809947] sched_clock: Marking stable (3809923333, 0)->(4773306277, -963382944) [ 3.813544] registered taskstats version 1 [ 3.815656] Loading compiled-in X.509 certificates [ 3.817757] zswap: loaded using pool lzo/zbud [ 3.842576] Key type big_key registered [ 3.854314] Key type encrypted registered [ 3.855996] ima: No TPM chip found, activating TPM-bypass! [ 3.858190] ima: Allocated hash algorithm: sha1 [ 3.860037] ima: No architecture policies found [ 3.861686] evm: Initialising EVM extended attributes: [ 3.863624] evm: security.selinux [ 3.864652] evm: security.ima [ 3.865619] evm: security.capability [ 3.866827] evm: HMAC attrs: 0x1 [ 3.870399] rtc_cmos 00:05: setting system clock to 2026-08-31 04:49:29 UTC (1788151769) [ 3.876958] debug: unmapping init [mem 0xffffffff9b803000-0xffffffff9b9fffff] [ 3.879712] debug: unmapping init [mem 0xffffffff9a582000-0xffffffff9a858fff] [ 3.887219] Write protecting the kernel read-only data: 28672k [ 3.890393] debug: unmapping init [mem 0xffffffff98c03000-0xffffffff98dfffff] [ 3.892849] debug: unmapping init [mem 0xffffffff99514000-0xffffffff995fffff] [ 3.929492] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.937544] systemd[1]: Detected virtualization kvm. [ 3.939595] systemd[1]: Detected architecture x86-64. [ 3.941681] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.971912] systemd[1]: No hostname configured. [ 3.973754] systemd[1]: Set hostname to . [ 3.975895] random: systemd: uninitialized urandom read (16 bytes read) [ 3.978079] systemd[1]: Initializing machine ID from random generator. [ 4.041231] random: ln: uninitialized urandom read (6 bytes read) [ 4.167895] random: systemd: uninitialized urandom read (16 bytes read) [ 4.170357] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 4.188862] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.194498] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Sockets. Starting Apply Kernel Variables... [ OK ] Reached target Swap. [ OK ] Reached target Paths. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.847095] device-mapper: uevent: version 1.0.3 [ 4.849158] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. [ 5.173226] random: fast init done Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.631592] virtio_net virtio0 ens2: renamed from eth0 [ 5.710338] scsi host0: ata_piix [ 5.753614] scsi host1: ata_piix [ 5.755384] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.757376] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 10.134598] dracut-initqueue[590]: RTNETLINK answers: File exists [ 10.373892] random: crng init done [ 10.375248] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.884551] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Sockets. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 12.092728] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.370524] SELinux: Disabled at runtime. [ 12.433637] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.443100] systemd[1]: Detected virtualization kvm. [ 12.445059] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.952981] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.956459] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.961646] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.965990] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.969785] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.980350] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.985665] systemd[1]: Listening on udev Kernel Socket. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. Starting Apply Kernel Variables... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Journal Service. [ OK ] Started Apply Kernel Variables.[ 13.195828] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 13.515037] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.822489] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.927606] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 14.037621] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 14.053891] EDAC sbridge: Ver: 1.1.2 [ 15.557417] Key type dns_resolver registered [ 15.869518] NFS: Registering the id_resolver key type [ 15.871566] Key type id_resolver registered [ 15.873216] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting System Logging Service... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg109-server login: [ 60.571512] libcfs: loading out-of-tree module taints kernel. [ 60.651778] Key type ._llcrypt registered [ 60.654744] Key type .llcrypt registered [ 60.758991] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 80.191622] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 81.697543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 81.729054] alg: No test for adler32 (adler32-zlib) [ 83.327355] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 84.452499] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 86.151220] Key type lgssc registered [ 87.824770] Lustre: Echo OBD driver; http://www.lustre.org/ [ 88.483205] hrtimer: interrupt took 2282190 ns [ 105.010888] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 155.358447] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 169.819419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 169.858590] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 171.173911] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 171.216231] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 171.380938] Lustre: lustre-MDT0000: new disk, initializing [ 171.542424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 171.579701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 177.029728] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 192.600890] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 192.742528] Lustre: 6512:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 192.806471] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 192.815175] Lustre: Skipped 1 previous similar message [ 192.949144] Lustre: lustre-MDT0001: new disk, initializing [ 193.052919] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 193.082745] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 193.091672] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 197.573985] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 201.854342] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 210.290031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 210.696596] Lustre: lustre-OST0000: new disk, initializing [ 210.699554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 210.703159] Lustre: 8450:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 210.787748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 215.120624] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 215.137212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 215.254767] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 217.425936] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 234.551061] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 234.722303] Lustre: lustre-OST0001: new disk, initializing [ 234.725845] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 234.732226] Lustre: 9523:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 234.821273] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 242.216081] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 242.227100] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 242.302193] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 243.191368] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.088940] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 265.141317] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 271.347699] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing check_logdir /tmp/testlogs/ [ 277.745790] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing yml_node [ 282.942529] Lustre: DEBUG MARKER: Client: 2.17.55.27 [ 285.380341] Lustre: DEBUG MARKER: MDS: 2.17.55.27 [ 287.864716] Lustre: DEBUG MARKER: OSS: 2.17.55.27 [ 289.429099] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 31 00:54:12 EDT 2026 [ 306.106228] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 307.933178] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 326.120145] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 326.131603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 326.139612] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.191554] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 329.193176] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 329.224460] Lustre: Skipped 2 previous similar messages [ 330.745384] Lustre: server umount lustre-MDT0000 complete [ 334.856528] LustreError: 7927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152100 with bad export cookie 3294581422675424312 [ 334.860320] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 334.878508] LustreError: 7927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 339.425794] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 339.429495] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 339.451380] LustreError: 6521:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 339.486400] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 339.497941] Lustre: Skipped 4 previous similar messages [ 341.524220] Lustre: server umount lustre-MDT0001 complete [ 346.826293] Lustre: server umount lustre-OST0000 complete [ 351.719264] Lustre: server umount lustre-OST0001 complete [ 360.354710] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 371.017608] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 411.522421] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 411.950913] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 412.021358] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 412.208257] Lustre: lustre-MDT0000: new disk, initializing [ 412.389272] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 412.427545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 418.583136] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 432.144525] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 432.363510] Lustre: 15082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 432.466617] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 432.470325] Lustre: Skipped 1 previous similar message [ 432.642604] Lustre: lustre-MDT0001: new disk, initializing [ 432.806543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 432.859556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 432.866234] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 438.561772] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 449.647622] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 455.609767] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 463.079178] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 463.429357] Lustre: lustre-OST0000: new disk, initializing [ 463.434871] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 463.444715] Lustre: 17053:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 463.596142] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 465.333439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 465.343352] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 465.502467] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 470.883855] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 483.227745] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 488.448564] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 488.750197] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 494.276743] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 494.595371] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 501.218237] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 501.240950] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 501.265451] Lustre: Skipped 1 previous similar message [ 501.288297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 502.985400] Lustre: server umount lustre-OST0000 complete [ 509.410257] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 509.416941] LustreError: Skipped 1 previous similar message [ 509.429587] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 509.452932] Lustre: Skipped 1 previous similar message [ 509.466939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 509.478893] Lustre: Skipped 1 previous similar message [ 513.893204] Lustre: server umount lustre-MDT0000 complete [ 516.582097] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 516.603215] LustreError: 15094:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 518.392288] LustreError: 17052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152284 with bad export cookie 3294581422675425845 [ 518.413158] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 518.416730] LustreError: 17052:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 518.949747] Lustre: server umount lustre-MDT0001 complete [ 523.330828] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 00:58:06 (1788152286) [ 526.079905] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 528.039305] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 00:58:11 (1788152291) [ 535.621069] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 536.227290] LustreError: 19462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 536.292347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 541.099173] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 541.676282] LustreError: 19463:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 545.766483] LustreError: 19462:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 549.693448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 555.258181] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 563.098091] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 569.703666] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 576.684566] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 583.799388] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 593.166357] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 604.651703] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 604.658980] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 604.670070] Lustre: Skipped 1 previous similar message [ 604.679650] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 604.691048] Lustre: Skipped 3 previous similar messages [ 610.487820] Lustre: server umount lustre-OST0000 complete [ 620.711269] Lustre: server umount lustre-MDT0000 complete [ 624.466401] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152390 with bad export cookie 3294581422675426727 [ 624.475983] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 624.477887] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 625.052795] Lustre: server umount lustre-MDT0001 complete [ 637.074906] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 639.947707] Key type lgssc unregistered [ 640.276804] LNet: 22995:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 640.289261] LNetError: 22995:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 640.310592] LNet: Removed LNI 192.168.201.109@tcp [ 641.290165] Key type .llcrypt unregistered [ 641.292086] Key type ._llcrypt unregistered [ 665.290481] Key type ._llcrypt registered [ 665.292396] Key type .llcrypt registered [ 665.373437] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 666.287978] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 666.391337] alg: No test for adler32 (adler32-zlib) [ 667.504471] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 667.726287] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 669.423255] Key type lgssc registered [ 670.345519] Lustre: Echo OBD driver; http://www.lustre.org/ [ 678.038457] loop: module loaded [ 678.109472] loop0: detected capacity change from 0 to 204800000 [ 678.387424] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.405517] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 678.804472] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 685.935097] loop0: detected capacity change from 0 to 204800000 [ 686.223631] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.236256] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 686.245247] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 691.847259] loop0: detected capacity change from 0 to 204800000 [ 697.942438] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 697.985928] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 699.320431] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 699.402519] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 699.494313] Lustre: lustre-MDT0000: new disk, initializing [ 699.566201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 699.579574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 704.023059] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 712.531034] loop1: detected capacity change from 0 to 204800000 [ 718.114765] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 718.532026] Lustre: lustre-OST07c6: new disk, initializing [ 718.537554] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 718.541053] Lustre: Skipped 1 previous similar message [ 718.546397] Lustre: 26198:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 718.649870] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 725.128293] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 726.215568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 726.230867] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 726.276638] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 731.092391] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 731.107367] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 731.120506] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 731.131890] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 731.141221] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 731.159467] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 746.225842] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 746.229919] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 746.235911] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 746.246041] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 746.259565] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 746.267804] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 750.905583] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 750.921162] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 750.930684] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 750.945262] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.951605] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 750.960430] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.973129] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 750.984852] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 750.990221] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 751.001108] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 751.011143] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 751.017989] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.252953] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 754.265955] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 754.281869] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 754.288251] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.330326] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 754.340389] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.354177] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 754.368289] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.372924] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 754.383127] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 754.415341] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 754.425604] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.595057] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 759.601132] Lustre: 25184:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 759.612979] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 759.623020] Lustre: 25184:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.632774] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 759.640858] Lustre: 25184:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.649044] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 759.664827] Lustre: 25184:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.675793] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 759.679871] Lustre: 25184:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 759.687661] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 759.693436] Lustre: 25184:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 770.987869] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 257 < left 278, rollback = 2 [ 770.994751] Lustre: 25185:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 770.999452] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 771.005693] Lustre: 25185:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 771.011669] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 771.025093] Lustre: 25185:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 771.037700] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 771.046442] Lustre: 25185:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 771.058782] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 771.072440] Lustre: 25185:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 771.084951] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 771.095806] Lustre: 25185:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 818.658764] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 818.671765] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 823.764175] Lustre: server umount lustre-OST07c6 complete [ 833.149920] Lustre: server umount lustre-MDT0000 complete [ 850.749300] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 854.186231] Key type lgssc unregistered [ 854.594709] LNet: 28751:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 854.599689] LNetError: 28751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 854.624253] LNet: Removed LNI 192.168.201.109@tcp [ 855.538869] Key type .llcrypt unregistered [ 855.541603] Key type ._llcrypt unregistered [ 876.658325] Key type ._llcrypt registered [ 876.660704] Key type .llcrypt registered [ 876.751435] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 877.774323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 877.903973] alg: No test for adler32 (adler32-zlib) [ 878.973498] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 879.212846] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 880.911222] Key type lgssc registered [ 882.221462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 891.606525] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 891.638330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 893.422583] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 899.459909] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 903.658428] LustreError: 30022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 903.686960] LustreError: 30022:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 908.786317] LustreError: 30023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 909.799390] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 910.304966] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 915.673701] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 925.877663] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 932.843507] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 940.247363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 940.621816] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.580136] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 954.164509] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 964.215634] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 979.940933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 979.944337] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 979.945597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 979.984179] Lustre: Skipped 1 previous similar message [ 985.067563] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 985.072364] Lustre: Skipped 1 previous similar message [ 986.129513] Lustre: server umount lustre-OST0000 complete [ 992.225825] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 992.233349] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 992.251039] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 992.256085] Lustre: Skipped 1 previous similar message [ 995.299539] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 995.301523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 995.319389] Lustre: Skipped 1 previous similar message [ 997.189596] Lustre: server umount lustre-MDT0000 complete [ 1000.420081] LustreError: 30023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1000.439084] LustreError: 30023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1001.506824] LustreError: 30004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152767 with bad export cookie 10521287598015070142 [ 1001.508783] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1001.515341] LustreError: 30004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1001.974315] Lustre: server umount lustre-MDT0001 complete [ 1013.804370] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1016.474931] Key type lgssc unregistered [ 1016.700938] LNet: 33783:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1016.705259] LNetError: 33783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1016.717812] LNet: Removed LNI 192.168.201.109@tcp [ 1017.393739] Key type .llcrypt unregistered [ 1017.395747] Key type ._llcrypt unregistered [ 1039.567355] Key type ._llcrypt registered [ 1039.571367] Key type .llcrypt registered [ 1039.713502] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1040.644781] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1040.677791] alg: No test for adler32 (adler32-zlib) [ 1041.866658] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1042.110045] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1043.847169] Key type lgssc registered [ 1045.365472] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1052.932764] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1052.949609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1054.519321] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1059.664942] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1064.929949] LustreError: 35059:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1064.957781] LustreError: 35059:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1069.128830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1069.558120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1074.374599] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1083.427934] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1090.488731] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1098.674479] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1099.079781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1100.168639] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1106.425357] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1114.255889] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1129.957574] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1129.964203] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1129.964962] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1129.973802] Lustre: Skipped 1 previous similar message [ 1132.443416] Lustre: server umount lustre-OST0000 complete [ 1136.103454] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1136.112263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1136.131914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1136.136422] Lustre: Skipped 1 previous similar message [ 1140.193431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1140.199569] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1142.472841] Lustre: server umount lustre-MDT0000 complete [ 1145.317943] LustreError: 35060:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1145.338078] LustreError: 35060:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1145.980523] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788152911 with bad export cookie 11107391420741972374 [ 1145.989968] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1145.993912] LustreError: 35041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1146.378522] Lustre: server umount lustre-MDT0001 complete [ 1159.599672] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1162.873852] Key type lgssc unregistered [ 1163.159808] LNet: 38593:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1163.167082] LNetError: 38593:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1163.196452] LNet: Removed LNI 192.168.201.109@tcp [ 1163.900636] Key type .llcrypt unregistered [ 1163.902515] Key type ._llcrypt unregistered [ 1192.412848] Key type ._llcrypt registered [ 1192.415206] Key type .llcrypt registered [ 1192.597140] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1194.043359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1194.336464] alg: No test for adler32 (adler32-zlib) [ 1195.549917] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1195.833447] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1197.584472] Key type lgssc registered [ 1199.296238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1207.350704] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1207.361503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1208.769167] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1214.012528] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1219.045273] LustreError: 40011:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1219.059603] LustreError: 40011:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1223.536877] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1223.787145] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1229.433665] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1237.898125] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1243.248991] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1250.372170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1250.606210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1253.672086] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1257.189852] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1265.072634] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1274.345939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1274.347044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1274.364600] Lustre: Skipped 1 previous similar message [ 1274.389691] Lustre: Skipped 2 previous similar messages [ 1277.347759] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 1279.457853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1279.475415] Lustre: Skipped 2 previous similar messages [ 1280.688194] Lustre: server umount lustre-MDT0000 complete [ 1282.417241] LustreError: 40006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.043034] LustreError: 39992:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153049 with bad export cookie 13432495127801182386 [ 1284.057183] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1284.576713] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1284.578363] LustreError: 40005:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1284.598173] Lustre: Skipped 1 previous similar message [ 1284.639738] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1287.534142] LustreError: 40006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1289.696660] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1289.699376] Lustre: Skipped 1 previous similar message [ 1292.653334] LustreError: 41172:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1292.666347] LustreError: 41172:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1297.785138] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [ 1297.790093] Lustre: Skipped 2 previous similar messages [ 1298.399468] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1298.604899] Lustre: server umount lustre-MDT0001 complete [ 1313.462740] Lustre: server umount lustre-OST0000 complete [ 1329.465410] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1332.515260] Key type lgssc unregistered [ 1332.854117] LNet: 43875:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.865665] LNetError: 43875:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.886944] LNet: Removed LNI 192.168.201.109@tcp [ 1333.890372] Key type .llcrypt unregistered [ 1333.892502] Key type ._llcrypt unregistered [ 1363.049344] Key type ._llcrypt registered [ 1363.051559] Key type .llcrypt registered [ 1363.150599] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1364.219672] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1364.349512] alg: No test for adler32 (adler32-zlib) [ 1365.464417] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1365.686386] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1367.359166] Key type lgssc registered [ 1368.450368] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1375.314767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1375.333240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1376.803907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1381.456837] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1386.982493] LustreError: 45295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1387.001867] LustreError: 45295:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1389.769086] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1390.042395] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1394.326212] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1401.977614] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1408.162740] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1415.989968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1416.552813] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1421.700243] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1424.835988] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1432.070981] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1442.278211] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1442.287051] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.313695] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.266470] Lustre: lustre-OST0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 1445.352606] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.374611] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1448.350114] Lustre: server umount lustre-OST0000 complete [ 1466.848384] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1466.861842] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1466.888146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1466.899875] Lustre: Skipped 2 previous similar messages [ 1470.948631] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1470.950977] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1473.044271] Lustre: server umount lustre-MDT0000 complete [ 1476.064515] LustreError: 45291:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1476.087896] LustreError: 45291:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1477.086338] LustreError: 45275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153242 with bad export cookie 14959641495802381462 [ 1477.106079] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1477.113586] LustreError: 45275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1483.530183] Lustre: server umount lustre-MDT0001 complete [ 1496.210336] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1499.434892] Key type lgssc unregistered [ 1499.778755] LNet: 49005:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1499.790282] LNetError: 49005:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1499.811357] LNet: Removed LNI 192.168.201.109@tcp [ 1501.043531] Key type .llcrypt unregistered [ 1501.046309] Key type ._llcrypt unregistered [ 1528.659657] Key type ._llcrypt registered [ 1528.663711] Key type .llcrypt registered [ 1528.803902] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1529.938498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1530.085130] alg: No test for adler32 (adler32-zlib) [ 1531.127151] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1531.322119] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1532.983406] Key type lgssc registered [ 1534.244747] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1541.770400] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1541.786354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1543.344282] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1548.149332] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1552.867705] LustreError: 50429:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1552.875550] LustreError: 50429:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1556.779843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1557.218621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1561.204895] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1568.698131] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1574.996661] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1582.341591] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1582.799358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1587.961345] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1589.179897] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1597.611519] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1607.631576] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1612.600796] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1626.017117] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-31 1h16m27s [ 1627.567211] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1631.110223] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 1632.200185] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 1633.252060] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1633.258708] Lustre: Skipped 2 previous similar messages [ 1633.259471] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1633.276583] Lustre: Skipped 3 previous similar messages [ 1635.993169] Lustre: server umount lustre-MDT0000 complete [ 1638.250514] LustreError: 50425:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1639.682672] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1640.420102] LustreError: 50424:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1640.446545] LustreError: 50424:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1644.399843] LustreError: 52269:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1645.994162] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1646.234582] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1646.755709] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1646.831276] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1647.804848] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1652.270696] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1657.284311] Lustre: DEBUG MARKER: Wait for df (36747) ... [ 1659.500358] Lustre: DEBUG MARKER: done [ 1663.971472] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1663.974972] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1663.991809] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1664.001250] Lustre: Skipped 4 previous similar messages [ 1667.554130] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1667.560239] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1668.585666] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1668.592696] Lustre: Skipped 2 previous similar messages [ 1678.306792] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1678.571182] Lustre: server umount lustre-OST0000 complete [ 1683.940770] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1683.945363] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1683.959702] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1683.979278] Lustre: Skipped 1 previous similar message [ 1683.986909] Lustre: Skipped 3 previous similar messages [ 1688.772205] Lustre: server umount lustre-MDT0000 complete [ 1691.112476] LustreError: 51156:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1691.143418] LustreError: 51156:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1692.392601] LustreError: 50410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153458 with bad export cookie 4292113082942450886 [ 1692.393536] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1692.403103] LustreError: 50410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1692.732255] Lustre: server umount lustre-MDT0001 complete [ 1703.617852] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1706.336636] Key type lgssc unregistered [ 1706.692916] LNet: 55356:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1706.697357] LNetError: 55356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1706.724109] LNet: Removed LNI 192.168.201.109@tcp [ 1707.631263] Key type .llcrypt unregistered [ 1707.634028] Key type ._llcrypt unregistered [ 1752.307679] Key type ._llcrypt registered [ 1752.316124] Key type .llcrypt registered [ 1752.422517] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1753.465732] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1753.507345] alg: No test for adler32 (adler32-zlib) [ 1754.596885] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1754.776739] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1756.415176] Key type lgssc registered [ 1757.697551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1766.824413] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1766.839362] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.188799] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1768.246632] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1768.475852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1773.148882] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1781.980635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.164259] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1782.213807] Lustre: 57568:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1782.472211] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1787.288416] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1796.400616] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1803.441415] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1810.622244] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1810.931964] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1811.224706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1817.234097] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1817.865671] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 1826.505942] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1833.034728] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-31 1h19m56s [ 1834.989288] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1843.818683] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1846.672409] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1846.831122] LustreError: 57574:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1867.194522] Lustre: lustre-MDT0000: Client 50ad093b-8083-4761-a627-a98c316f8f03 (at 192.168.201.9@tcp) reconnecting [ 1886.863324] LustreError: 57574:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1890.407728] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1893.982941] Lustre: DEBUG MARKER: done [ 1904.129690] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1904.147464] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1904.168950] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1909.446908] Lustre: server umount lustre-OST0000 complete [ 1914.849756] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1914.851170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1914.852863] Lustre: Skipped 2 previous similar messages [ 1914.863296] Lustre: Skipped 2 previous similar messages [ 1915.873740] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1915.887342] Lustre: Skipped 1 previous similar message [ 1919.997524] Lustre: server umount lustre-MDT0000 complete [ 1924.376306] LustreError: 57560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153690 with bad export cookie 12114593909102100158 [ 1924.389113] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1924.394172] LustreError: 57560:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1924.894738] Lustre: server umount lustre-MDT0001 complete [ 1937.711991] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 1940.712184] Key type lgssc unregistered [ 1941.039488] LNet: 61898:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1941.048448] LNetError: 61898:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1941.072530] LNet: Removed LNI 192.168.201.109@tcp [ 1941.900695] Key type .llcrypt unregistered [ 1941.902641] Key type ._llcrypt unregistered [ 1991.331269] Key type ._llcrypt registered [ 1991.333531] Key type .llcrypt registered [ 1991.445527] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 1992.476860] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1992.589137] alg: No test for adler32 (adler32-zlib) [ 1993.757627] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 1994.064841] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 1995.775189] Key type lgssc registered [ 1997.273139] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2001.051981] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2001.067744] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2002.441745] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2002.457475] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2002.641867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2003.657375] Lustre: Failing over lustre-MDT0000 [ 2003.898991] Lustre: server umount lustre-MDT0000 complete [ 2012.073753] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 01:22:55 (1788153775) [ 2019.750852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2020.394810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2025.837612] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2036.690651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2036.961786] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2037.060410] Lustre: 64698:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2037.465696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2044.624287] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2053.234409] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2060.561703] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2069.250792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2069.538632] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2069.949622] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2077.143562] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2078.472108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2085.692768] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2091.224805] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2099.168312] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2099.172756] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2099.178732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2099.208521] Lustre: Skipped 2 previous similar messages [ 2104.289297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2104.297868] Lustre: Skipped 4 previous similar messages [ 2105.314868] Lustre: server umount lustre-MDT0000 complete [ 2109.130785] LustreError: 65436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153874 with bad export cookie 14463414117229667031 [ 2109.133233] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2109.136795] LustreError: 65436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2109.411654] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2109.523791] Lustre: server umount lustre-MDT0001 complete [ 2113.397383] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2119.083699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2128.565747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2133.985831] LustreError: 63620:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000bc921f24 x1875015116040192/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2134.294379] LustreError: 66552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2134.313772] LustreError: 66552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2134.387117] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2134.465922] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2138.301424] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2139.490693] LustreError: 66553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2139.494068] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2144.739200] LustreError: 66552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2146.789933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2147.064062] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2150.120463] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2151.652726] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2161.543306] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2169.932197] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2181.100469] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2181.112058] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2181.119270] Lustre: Skipped 2 previous similar messages [ 2181.130092] Lustre: Skipped 2 previous similar messages [ 2185.040864] Lustre: server umount lustre-MDT0000 complete [ 2186.221779] LustreError: 69582:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2186.253815] LustreError: 69582:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2188.593328] LustreError: 64691:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788153954 with bad export cookie 14463414117229669313 [ 2188.599722] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2188.873577] Lustre: server umount lustre-MDT0001 complete [ 2195.981490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2206.498315] Lustre: 63622:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788153956/real 1788153956] req@0000000059d781a3 x1875015116068608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788153972 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2206.513432] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2207.164470] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2209.759287] Lustre: 63624:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788153959/real 1788153959] req@0000000059ed8a2b x1875015116068736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788153975 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2212.831664] Lustre: 63623:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788153962/real 1788153962] req@0000000081fee45d x1875015116068864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788153978 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2214.217293] LustreError: 66552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2214.304764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2214.368721] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2214.882271] Lustre: 63621:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788153964/real 1788153964] req@00000000996e0fa8 x1875015116069120/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788153980 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2219.627286] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2220.066695] Lustre: 63621:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788153969/real 1788153969] req@00000000c54bbdb4 x1875015116069248/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788153985 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2222.573245] LustreError: 71154:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2222.593434] LustreError: 71154:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2229.277087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2234.832859] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2234.866837] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2243.754163] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2252.592764] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2265.570879] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2265.573398] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2265.604282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2271.726668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2271.737793] Lustre: Skipped 5 previous similar messages [ 2276.319158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2276.707058] Lustre: server umount lustre-MDT0000 complete [ 2276.845719] LustreError: 71155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2276.873417] LustreError: 71155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2281.629160] LustreError: 65436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154047 with bad export cookie 14463414117229670692 [ 2281.635952] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2281.643560] LustreError: 65436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 2281.953490] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2281.966359] Lustre: Skipped 2 previous similar messages [ 2282.062084] Lustre: server umount lustre-MDT0001 complete [ 2289.380625] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2293.596964] Lustre: DEBUG MARKER: files compared the same [ 2296.135852] Lustre: server umount lustre-OST0000 complete [ 2315.049411] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2318.099366] Key type lgssc unregistered [ 2318.499815] LNet: 74432:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2318.504389] LNetError: 74432:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2318.515984] LNet: Removed LNI 192.168.201.109@tcp [ 2319.318277] Key type .llcrypt unregistered [ 2319.320533] Key type ._llcrypt unregistered [ 2347.174926] Key type ._llcrypt registered [ 2347.179800] Key type .llcrypt registered [ 2347.320630] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2348.906038] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2348.927198] alg: No test for adler32 (adler32-zlib) [ 2350.071306] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2350.360169] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2352.143209] Key type lgssc registered [ 2353.375311] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2361.535562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2361.570167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2363.071753] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2367.008684] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug +malloc all [ 2372.583701] LustreError: 75853:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.603527] LustreError: 75853:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2374.925139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2375.235733] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2380.125571] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug +malloc all [ 2388.259468] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2394.496179] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2402.270896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2402.715725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2407.864825] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2408.683884] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug +malloc all [ 2416.015666] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2431.458450] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2431.462404] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2431.466420] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2431.479314] Lustre: Skipped 1 previous similar message [ 2434.981804] Lustre: server umount lustre-OST0000 complete [ 2441.705979] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2441.707903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2441.708691] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2441.730181] Lustre: Skipped 1 previous similar message [ 2441.749856] Lustre: Skipped 2 previous similar messages [ 2445.993419] Lustre: server umount lustre-MDT0000 complete [ 2446.817165] LustreError: 78343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.837159] LustreError: 78343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2449.560868] LustreError: 75833:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154215 with bad export cookie 13088614441229559033 [ 2449.572317] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2449.869994] Lustre: server umount lustre-MDT0001 complete [ 2463.134799] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2465.958583] Key type lgssc unregistered [ 2466.311839] LNet: 79414:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2466.316622] LNetError: 79414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2467.369172] LNet: Removed LNI 192.168.201.109@tcp [ 2468.353111] Key type .llcrypt unregistered [ 2468.355899] Key type ._llcrypt unregistered [ 2496.144441] Key type ._llcrypt registered [ 2496.146798] Key type .llcrypt registered [ 2496.301383] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2497.662368] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2497.753105] alg: No test for adler32 (adler32-zlib) [ 2498.851670] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2499.052275] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2500.727191] Key type lgssc registered [ 2502.080359] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2509.897715] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2509.941393] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2525.471732] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2530.591416] LustreError: 80805:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [ 2557.279274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2564.171341] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2573.006152] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2582.612526] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2583.193050] Lustre: *** cfs_fail_loc=706, val=0*** [ 2584.352193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2584.534682] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2589.675150] LustreError: 80830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2589.694290] LustreError: 80830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2590.831980] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2591.743276] LustreError: 81889:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2591.798203] LustreError: 81889:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2596.838544] LustreError: 80830:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2596.885156] LustreError: 80830:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2601.957931] LustreError: 80831:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2601.991416] LustreError: 80831:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2604.068073] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2604.595679] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2611.265422] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2621.833647] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2631.160633] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2635.240107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2635.257256] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2635.262698] Lustre: Skipped 1 previous similar message [ 2635.271434] Lustre: Skipped 1 previous similar message [ 2640.138803] Lustre: server umount lustre-OST0000 complete [ 2645.473876] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2645.477036] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2645.485489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2645.496358] Lustre: Skipped 1 previous similar message [ 2650.593681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2650.603028] Lustre: Skipped 1 previous similar message [ 2654.689553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2654.700274] Lustre: Skipped 1 previous similar message [ 2658.271541] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2658.496620] Lustre: server umount lustre-MDT0000 complete [ 2659.808484] LustreError: 81888:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.828312] LustreError: 81888:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 2662.012251] LustreError: 80812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154427 with bad export cookie 6142302258036098587 [ 2662.023850] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2662.027329] LustreError: 80812:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2662.662603] Lustre: server umount lustre-MDT0001 complete [ 2674.822239] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 2677.935273] Key type lgssc unregistered [ 2678.258303] LNet: 84424:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2678.269079] LNetError: 84424:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2678.296180] LNet: Removed LNI 192.168.201.109@tcp [ 2679.096939] Key type .llcrypt unregistered [ 2679.099024] Key type ._llcrypt unregistered [ 2711.273946] Key type ._llcrypt registered [ 2711.276443] Key type .llcrypt registered [ 2711.379428] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2712.959488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2713.037814] alg: No test for adler32 (adler32-zlib) [ 2714.102844] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 2714.285904] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 2715.935276] Key type lgssc registered [ 2717.761225] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2728.518892] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2728.532474] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2735.075396] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2744.181852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2744.850675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2748.985372] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2749.931721] LustreError: 86535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2749.946857] LustreError: 86535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2755.051868] LustreError: 86539:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2757.488205] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2757.872303] LustreError: 86535:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2757.908926] LustreError: 86535:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2757.925384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2763.237350] LustreError: 86534:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2763.263502] LustreError: 86534:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2763.907140] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2768.353961] LustreError: 86538:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2768.384754] LustreError: 86538:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 2772.762979] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2772.782480] Lustre: MGS: Client ca515c2b-03d1-4eb8-b1bc-99f4bb353106 (at 0@lo) reconnecting [ 2772.826032] LustreError: 86549:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2773.109370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.208336] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2777.860417] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2793.445905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2793.463945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2793.475187] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2798.562532] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2798.582217] Lustre: Skipped 2 previous similar messages [ 2799.189883] Lustre: server umount lustre-OST0000 complete [ 2803.685850] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2803.692920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2803.706031] Lustre: Skipped 1 previous similar message [ 2803.707127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2808.800815] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2808.812194] Lustre: Skipped 2 previous similar messages [ 2808.910234] Lustre: server umount lustre-MDT0000 complete [ 2813.586961] Lustre: server umount lustre-MDT0001 complete [ 2817.531075] Lustre: server umount MGS complete [ 2829.689405] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 01:36:32 (1788154592) [ 2859.252890] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 2870.303285] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 2908.260708] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2913.723327] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2925.412699] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2925.659312] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2925.707705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2925.842961] Lustre: lustre-MDT0001: new disk, initializing [ 2926.066615] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2926.130454] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2926.139652] Lustre: 93722:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2926.145503] LustreError: 93722:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2930.443186] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2940.958944] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2941.302903] Lustre: lustre-OST0000: new disk, initializing [ 2941.307692] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2941.322754] Lustre: 94487:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2941.412651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2947.676884] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2947.690305] Lustre: 94778:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000059ed8a2b x1875015871013504/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788154729 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2958.080054] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2965.594108] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2965.604467] Lustre: MGS: Client ad03b2f4-f765-4b6f-9168-b5b18f1bbdbe (at 0@lo) reconnecting [ 2965.645284] Lustre: 95532:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2965.657571] LustreError: 93708:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2965.670583] LustreError: 93708:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2965.684859] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2965.740660] Lustre: lustre-MDT0000: new disk, initializing [ 2965.891746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2965.919216] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2965.953732] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2967.009539] Lustre: 95556:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000070037e0 x1875015871020416/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788154748 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2970.622953] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 2972.129403] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2975.236582] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 2975.254774] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 2976.107317] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 2976.184479] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3004.392097] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3004.403030] Lustre: Skipped 2 previous similar messages [ 3004.406769] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3004.417807] Lustre: Skipped 1 previous similar message [ 3014.638850] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3014.650118] Lustre: Skipped 3 previous similar messages [ 3017.183225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3017.342158] Lustre: server umount lustre-OST0000 complete [ 3024.866118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3024.868933] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.387441] Lustre: server umount lustre-MDT0000 complete [ 3029.986646] LustreError: 95994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3030.013352] LustreError: 95994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3046.367364] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3046.541215] Lustre: server umount lustre-MDT0001 complete [ 3051.243624] Lustre: server umount MGS complete [ 3063.487501] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 01:40:26 (1788154826) [ 3083.935828] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3086.871476] Key type lgssc unregistered [ 3087.246918] LNet: 98063:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3087.252153] LNetError: 98063:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3087.270293] LNet: Removed LNI 192.168.201.109@tcp [ 3088.255739] Key type .llcrypt unregistered [ 3088.260723] Key type ._llcrypt unregistered [ 3111.013649] Key type ._llcrypt registered [ 3111.016819] Key type .llcrypt registered [ 3111.163721] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3112.000705] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3112.023154] alg: No test for adler32 (adler32-zlib) [ 3113.152373] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3113.486034] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3115.119612] Key type lgssc registered [ 3116.594750] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3127.050843] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3127.075781] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3128.754821] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3135.963369] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3139.052184] LustreError: 99285:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3139.072220] LustreError: 99285:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3144.180857] LustreError: 99286:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.470392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3148.177259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3152.805863] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3160.084711] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3166.839878] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3174.129798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3174.615037] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3175.659545] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3181.744525] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3190.146905] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3203.560561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3203.581258] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3203.597963] Lustre: Skipped 1 previous similar message [ 3208.674194] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3208.679025] Lustre: Skipped 1 previous similar message [ 3209.568372] Lustre: server umount lustre-OST0000 complete [ 3213.796324] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3213.799115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3213.830261] Lustre: Skipped 2 previous similar messages [ 3213.847934] Lustre: Skipped 1 previous similar message [ 3218.914106] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3218.919655] Lustre: Skipped 1 previous similar message [ 3219.837963] Lustre: server umount lustre-MDT0000 complete [ 3223.382780] LustreError: 99267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788154989 with bad export cookie 11599089921825573450 [ 3223.388655] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3223.405140] LustreError: 99267:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3223.870959] Lustre: server umount lustre-MDT0001 complete [ 3235.644719] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3238.346657] Key type lgssc unregistered [ 3238.674712] LNet: 102837:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3238.683295] LNetError: 102837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3238.698559] LNet: Removed LNI 192.168.201.109@tcp [ 3239.678326] Key type .llcrypt unregistered [ 3239.681395] Key type ._llcrypt unregistered [ 3257.939197] Key type ._llcrypt registered [ 3257.945425] Key type .llcrypt registered [ 3258.047602] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3259.251184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3259.412085] alg: No test for adler32 (adler32-zlib) [ 3260.818480] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3260.995353] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3262.631451] Key type lgssc registered [ 3263.614366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3272.055870] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3272.063978] LustreError: 103923:0:(tgt_mount.c:2644:server_fill_super()) cfs_race id 716 sleeping [ 3272.074516] LustreError: 103922:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 waking [ 3272.086428] LustreError: 103923:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3272.099626] LustreError: 103923:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3272.106161] LustreError: 103923:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3272.167442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3273.705366] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3283.939206] LustreError: 103960:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.999387] LustreError: 103960:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3284.929651] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3285.540318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3291.617148] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3298.149943] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lsmod [ 3301.471049] LustreError: 105126:0:(tgt_mount.c:2644:server_fill_super()) cfs_race id 716 sleeping [ 3301.501288] LustreError: 105129:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 waking [ 3301.510753] LustreError: 105126:0:(tgt_mount.c:2644:server_fill_super()) cfs_fail_race id 716 awake: rc=4976 [ 3301.523969] LustreError: 105126:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3301.531555] LustreError: 105126:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3301.533847] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3302.051286] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3307.501686] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3307.539227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3307.545284] Lustre: Skipped 1 previous similar message [ 3310.630155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3310.652780] Lustre: Skipped 4 previous similar messages [ 3312.982704] Lustre: server umount lustre-MDT0000 complete [ 3315.695030] LustreError: 103956:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3315.711012] LustreError: 103956:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3317.986763] LustreError: 103941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155083 with bad export cookie 7540219802496179621 [ 3318.017075] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3318.943878] LustreError: 104392:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000008f3c8a23 x1875016444565248/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3319.224654] Lustre: server umount lustre-MDT0001 complete [ 3333.877667] Lustre: server umount lustre-OST0000 complete [ 3343.026262] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3343.422410] LustreError: 106195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3343.535953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3348.961766] LustreError: 106196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3349.065498] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3354.087960] LustreError: 106195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3358.831687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3359.076797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3364.139264] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3373.077758] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3380.990697] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3388.286873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3388.539698] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3389.571128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3394.886261] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3402.566117] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3409.380833] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3409.391275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3409.412075] Lustre: Skipped 1 previous similar message [ 3409.423388] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3409.436051] Lustre: Skipped 1 previous similar message [ 3411.088931] Lustre: server umount lustre-OST0000 complete [ 3415.520141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3415.523668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3415.538301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.620550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3419.624181] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3421.003094] Lustre: server umount lustre-MDT0000 complete [ 3424.732648] LustreError: 106177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155190 with bad export cookie 7540219802496180321 [ 3424.741902] LustreError: 106177:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3424.750692] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3425.148823] Lustre: server umount lustre-MDT0001 complete [ 3438.267939] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3440.509431] Key type lgssc unregistered [ 3440.766085] LNet: 109728:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3440.778726] LNetError: 109728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3440.799718] LNet: Removed LNI 192.168.201.109@tcp [ 3441.579515] Key type .llcrypt unregistered [ 3441.581992] Key type ._llcrypt unregistered [ 3465.578790] Key type ._llcrypt registered [ 3465.580451] Key type .llcrypt registered [ 3465.676973] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3466.729503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3466.814407] alg: No test for adler32 (adler32-zlib) [ 3467.902877] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3468.129616] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3469.791355] Key type lgssc registered [ 3471.089306] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3478.664295] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3478.682088] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3480.192068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3484.518219] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3489.777240] LustreError: 111145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3489.814629] LustreError: 111145:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3493.544376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3493.843937] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3498.387223] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3507.077770] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3513.491749] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3521.621285] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3522.059537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3525.113021] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:67 to 0x240000401:97) [ 3530.629052] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3540.004664] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3546.121643] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3549.296497] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3554.785626] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3554.796042] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3554.808373] Lustre: Skipped 1 previous similar message [ 3554.808726] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3554.833774] Lustre: Skipped 1 previous similar message [ 3559.536661] Lustre: server umount lustre-OST0000 complete [ 3565.041243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3565.055951] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3565.064901] Lustre: Skipped 1 previous similar message [ 3565.106934] Lustre: Skipped 1 previous similar message [ 3569.519052] Lustre: server umount lustre-MDT0000 complete [ 3570.149658] LustreError: 111142:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.171833] LustreError: 111142:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3573.960328] LustreError: 111127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155339 with bad export cookie 15171738877022920387 [ 3573.963514] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3573.974063] LustreError: 111127:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3574.235810] Lustre: server umount lustre-MDT0001 complete [ 3588.997451] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3592.454685] Key type lgssc unregistered [ 3592.830733] LNet: 114784:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3592.837803] LNetError: 114784:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3592.855379] LNet: Removed LNI 192.168.201.109@tcp [ 3594.239146] Key type .llcrypt unregistered [ 3594.249188] Key type ._llcrypt unregistered [ 3615.448666] Key type ._llcrypt registered [ 3615.450649] Key type .llcrypt registered [ 3615.609854] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3617.194908] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3617.244390] alg: No test for adler32 (adler32-zlib) [ 3618.637652] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3618.888186] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3620.567772] Key type lgssc registered [ 3621.982478] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3630.466669] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3630.505945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3632.072580] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3638.189587] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3642.338730] LustreError: 116006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3642.348214] LustreError: 116006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3647.478687] LustreError: 116007:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3648.992949] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3649.455416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3654.200351] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3663.564624] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3670.687930] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3680.411894] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3680.995475] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3690.314231] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3701.187465] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3701.442909] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3711.968856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3711.985614] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3711.990818] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3712.022761] Lustre: Skipped 1 previous similar message [ 3717.093629] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3717.105559] Lustre: Skipped 2 previous similar messages [ 3717.364556] Lustre: server umount lustre-OST0000 complete [ 3722.215298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3722.229123] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3722.247766] Lustre: Skipped 1 previous similar message [ 3722.280833] Lustre: Skipped 1 previous similar message [ 3727.329192] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3727.335469] Lustre: Skipped 1 previous similar message [ 3728.369578] Lustre: server umount lustre-MDT0000 complete [ 3732.452255] LustreError: 116006:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3732.471513] LustreError: 116006:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3733.030037] LustreError: 115988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155498 with bad export cookie 8779287457978597564 [ 3733.036340] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3733.042109] LustreError: 115988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3733.343742] Lustre: server umount lustre-MDT0001 complete [ 3748.819147] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3752.282666] Key type lgssc unregistered [ 3752.580109] LNet: 119548:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3752.602109] LNetError: 119548:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3752.621768] LNet: Removed LNI 192.168.201.109@tcp [ 3753.594486] Key type .llcrypt unregistered [ 3753.597764] Key type ._llcrypt unregistered [ 3784.228220] Key type ._llcrypt registered [ 3784.230645] Key type .llcrypt registered [ 3784.326618] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 3786.183224] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3786.291221] alg: No test for adler32 (adler32-zlib) [ 3787.534658] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 3787.906179] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 3789.719194] Key type lgssc registered [ 3790.932438] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3802.336989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3802.389780] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3803.926539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3809.133345] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3814.385257] LustreError: 120969:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3814.410255] LustreError: 120969:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3819.503934] LustreError: 120968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3819.644832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3820.219964] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3826.907662] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3836.738902] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3843.506402] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3851.985490] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3852.452341] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3858.304624] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 3867.027698] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3875.029950] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3885.142790] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3889.705781] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3889.713538] Lustre: Skipped 2 previous similar messages [ 3893.856660] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3893.880799] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3895.582886] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3895.592217] Lustre: Skipped 2 previous similar messages [ 3902.102399] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3902.112885] Lustre: Skipped 1 previous similar message [ 3910.989875] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.9@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3910.996135] Lustre: Skipped 2 previous similar messages [ 3912.119666] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.9@tcp 192.168.0.[2,10]@tcp [ 3923.882584] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.9/24@tcp in log lustre-MDT0000 [ 3923.890219] Lustre: Skipped 2 previous similar messages [ 3931.803662] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.9/24@tcp [ 3931.821672] Lustre: Skipped 1 previous similar message [ 3939.811822] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3939.814323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3939.840824] Lustre: Skipped 1 previous similar message [ 3939.852075] Lustre: Skipped 1 previous similar message [ 3941.300239] Lustre: server umount lustre-OST0000 complete [ 3948.513432] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3948.522900] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3948.543064] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3950.048025] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3950.049929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3951.994408] Lustre: server umount lustre-MDT0000 complete [ 3955.175709] LustreError: 122045:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3955.203536] LustreError: 122045:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3956.270257] LustreError: 120949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155721 with bad export cookie 8767977255557603929 [ 3956.289408] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3956.300561] LustreError: 120949:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3956.909486] Lustre: server umount lustre-MDT0001 complete [ 3968.760466] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 3971.713866] Key type lgssc unregistered [ 3971.970917] LNet: 125890:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3971.979549] LNetError: 125890:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3972.004353] LNet: Removed LNI 192.168.201.109@tcp [ 3972.873441] Key type .llcrypt unregistered [ 3972.875071] Key type ._llcrypt unregistered [ 4005.888018] Key type ._llcrypt registered [ 4005.893856] Key type .llcrypt registered [ 4006.014224] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4006.924915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4007.020855] alg: No test for adler32 (adler32-zlib) [ 4008.130273] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4008.359160] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4010.023178] Key type lgssc registered [ 4011.231576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4018.916307] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4018.931794] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4020.433948] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4020.454643] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.9/24@tcp [ 4020.486986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4025.206195] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4030.948362] LustreError: 127454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4030.966622] LustreError: 127454:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4034.658695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4035.050454] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4035.085418] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.9/24@tcp [ 4035.139468] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4040.583747] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4048.879564] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4055.597298] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4063.157678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4063.468967] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4065.529719] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:169 to 0x240000401:193) [ 4069.775651] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4078.013945] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4088.805343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4088.817454] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4088.835643] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4090.338468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4090.344277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4090.364801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4095.077554] Lustre: server umount lustre-OST0000 complete [ 4100.577618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4100.594161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4100.596482] Lustre: Skipped 1 previous similar message [ 4105.537635] Lustre: server umount lustre-MDT0000 complete [ 4105.705165] LustreError: 127454:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4110.200229] LustreError: 127434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788155875 with bad export cookie 11714221904555847813 [ 4110.212942] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4110.217223] LustreError: 127434:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4110.614507] Lustre: server umount lustre-MDT0001 complete [ 4126.532245] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4130.247960] Key type lgssc unregistered [ 4130.497775] LNet: 131044:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4130.503059] LNetError: 131044:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4130.523776] LNet: Removed LNI 192.168.201.109@tcp [ 4131.527345] Key type .llcrypt unregistered [ 4131.529282] Key type ._llcrypt unregistered [ 4161.104619] Key type ._llcrypt registered [ 4161.106778] Key type .llcrypt registered [ 4161.202182] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4179.421812] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4181.324094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4181.407017] alg: No test for adler32 (adler32-zlib) [ 4182.528909] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4182.751772] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4184.407940] Key type lgssc registered [ 4185.617956] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4224.011806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4224.042695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4225.460909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4225.505857] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4225.615222] Lustre: lustre-MDT0000: new disk, initializing [ 4225.711782] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4225.733900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4230.917710] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4242.525463] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4242.666586] Lustre: 134449:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4242.717742] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4242.722538] Lustre: Skipped 1 previous similar message [ 4242.787511] Lustre: lustre-MDT0001: new disk, initializing [ 4242.885868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4242.923344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4242.936616] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4248.026889] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4258.969979] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4265.947695] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4273.596153] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4273.834927] Lustre: lustre-OST0000: new disk, initializing [ 4273.838583] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4273.843970] Lustre: 136415:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4273.940565] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4281.668956] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4282.431305] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4282.436932] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4282.592040] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4292.136962] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4297.081782] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4297.324096] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4301.457478] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4301.655967] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4308.454943] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4308.456300] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4308.460328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4308.484980] Lustre: Skipped 1 previous similar message [ 4313.570763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4313.586772] Lustre: Skipped 2 previous similar messages [ 4318.175229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4318.423761] Lustre: server umount lustre-OST0000 complete [ 4323.811783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4323.813277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4323.833584] Lustre: Skipped 1 previous similar message [ 4323.850696] Lustre: Skipped 1 previous similar message [ 4328.930532] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4328.946109] Lustre: Skipped 1 previous similar message [ 4329.517604] Lustre: server umount lustre-MDT0000 complete [ 4334.420429] LustreError: 134441:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156100 with bad export cookie 11153409490599939067 [ 4334.447915] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4334.888564] Lustre: server umount lustre-MDT0001 complete [ 4344.369458] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4344.950532] LustreError: 138647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.981633] LustreError: 138647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4345.041088] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4349.801631] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4350.434699] LustreError: 138647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4350.460509] LustreError: 138647:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4354.529983] LustreError: 138648:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.512753] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4359.656524] LustreError: 138647:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4360.207157] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4364.809844] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4373.250804] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4379.605887] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4386.962252] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4387.427388] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4393.765576] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4401.648125] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4406.851146] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4407.163891] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4412.200219] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4412.421291] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4421.373170] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4421.556939] Lustre: lustre-OST0001: new disk, initializing [ 4421.562933] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4421.569407] Lustre: 142019:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4421.637904] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4426.893951] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4426.904957] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4427.045886] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4428.612395] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4438.697303] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4438.859950] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4444.863582] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4445.176727] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4451.856535] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e2e080d6800.ost_server_uuid 50 [ 4453.986421] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e2e080d6800.ost_server_uuid in IDLE state after 0 sec [ 4463.084811] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4463.087897] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4463.091137] LustreError: Skipped 1 previous similar message [ 4463.092525] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4463.122305] Lustre: Skipped 1 previous similar message [ 4464.534522] Lustre: server umount lustre-OST0001 complete [ 4468.210252] LustreError: 140470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4468.266869] LustreError: 140470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4471.777559] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4471.779924] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4471.790394] LustreError: Skipped 2 previous similar messages [ 4471.801835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4471.835480] Lustre: Skipped 1 previous similar message [ 4471.835882] Lustre: Skipped 1 previous similar message [ 4478.446938] LustreError: 140470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4478.474970] LustreError: 140470:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4483.039251] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4483.265666] Lustre: server umount lustre-OST0000 complete [ 4488.164039] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4488.196853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4488.213210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4488.222378] Lustre: Skipped 5 previous similar messages [ 4493.987617] Lustre: server umount lustre-MDT0000 complete [ 4498.507582] LustreError: 138629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156264 with bad export cookie 11153409490599939949 [ 4498.511180] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4498.524968] LustreError: 138629:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 4499.003162] Lustre: server umount lustre-MDT0001 complete [ 4520.409876] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4523.766235] Key type lgssc unregistered [ 4524.109755] LNet: 145024:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4524.117837] LNetError: 145024:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4524.144812] LNet: Removed LNI 192.168.201.109@tcp [ 4525.188819] Key type .llcrypt unregistered [ 4525.190928] Key type ._llcrypt unregistered [ 4569.648754] Key type ._llcrypt registered [ 4569.651405] Key type .llcrypt registered [ 4569.731666] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4588.105624] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4589.547497] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4589.654505] alg: No test for adler32 (adler32-zlib) [ 4590.779472] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4590.986367] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4592.703190] Key type lgssc registered [ 4593.953788] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4633.460874] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4633.488448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4634.881402] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4634.928914] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4635.062799] Lustre: lustre-MDT0000: new disk, initializing [ 4635.190133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4635.220035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4639.926073] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4652.510299] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4652.683947] Lustre: 149189:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4652.781716] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4652.784585] Lustre: Skipped 1 previous similar message [ 4652.934970] Lustre: lustre-MDT0001: new disk, initializing [ 4653.085403] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4653.159879] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4653.225264] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4658.368654] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4669.258716] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4676.271316] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4683.845072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4684.228830] Lustre: lustre-OST0000: new disk, initializing [ 4684.236808] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4684.247310] Lustre: 151160:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4684.392327] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4690.916685] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4691.544298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4691.559695] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4691.751839] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4701.075523] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4706.970732] Lustre: Failing over lustre-OST0000 [ 4707.273086] Lustre: server umount lustre-OST0000 complete [ 4722.594104] Lustre: 147414:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156472/real 1788156472] req@00000000b89ca7c9 x1875017839184000/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788156488 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4722.630125] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4724.706496] Lustre: 147414:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156474/real 1788156474] req@000000008807454c x1875017839184384/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788156490 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4724.748701] Lustre: 147414:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4728.320295] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4728.751143] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4728.780193] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4728.792622] Lustre: 147413:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156478/real 1788156478] req@000000007ba5512d x1875017839185280/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788156494 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4732.860699] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4733.921596] Lustre: 147415:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156483/real 1788156483] req@00000000296420dd x1875017839186432/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788156499 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4733.958570] Lustre: 147415:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4735.916124] Lustre: lustre-OST0000: Recovery over after 0:03, of 3 clients 3 recovered and 0 were evicted. [ 4735.917960] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4736.511665] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4741.009644] Lustre: Failing over lustre-MDT0000 [ 4741.508210] Lustre: server umount lustre-MDT0000 complete [ 4743.139407] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.154833] Lustre: Skipped 2 previous similar messages [ 4748.256498] LustreError: 149195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4748.304052] LustreError: 149195:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 4751.212899] LustreError: 150228:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.380117] LustreError: 150389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4753.403698] LustreError: 150389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4756.338185] LustreError: 149195:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4758.495198] Lustre: 147412:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788156508/real 1788156508] req@0000000068439485 x1875017839198080/t0(0) o400->MGC192.168.201.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788156524 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4758.517938] Lustre: 147412:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4758.523461] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4761.349043] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4761.454319] LustreError: 149196:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4761.477193] LustreError: 149196:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 4768.740274] LustreError: 147411:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003c300105 x1875017839203584/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4769.134817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4769.196762] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4771.713475] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4774.381921] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4774.398805] Lustre: Skipped 1 previous similar message [ 4774.420553] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4774.476535] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4774.979543] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4784.610919] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4784.625055] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4784.646994] Lustre: Skipped 2 previous similar messages [ 4784.656277] Lustre: Skipped 1 previous similar message [ 4789.732375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4789.741250] Lustre: Skipped 1 previous similar message [ 4790.803365] Lustre: server umount lustre-OST0000 complete [ 4799.972086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4799.974783] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4799.983488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4800.009790] Lustre: Skipped 1 previous similar message [ 4801.676385] Lustre: server umount lustre-MDT0000 complete [ 4805.091496] LustreError: 150389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4805.104790] LustreError: 150389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 4805.343904] LustreError: 149180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788156570 with bad export cookie 14516448634063219188 [ 4805.358202] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4805.364041] LustreError: 149180:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4805.768227] Lustre: server umount lustre-MDT0001 complete [ 4824.154893] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 4829.256658] Key type lgssc unregistered [ 4829.669720] LNet: 154841:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4829.679420] LNetError: 154841:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4829.699527] LNet: Removed LNI 192.168.201.109@tcp [ 4830.772118] Key type .llcrypt unregistered [ 4830.776925] Key type ._llcrypt unregistered [ 4876.800688] Key type ._llcrypt registered [ 4876.814061] Key type .llcrypt registered [ 4876.964348] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4877.998893] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4878.079221] alg: No test for adler32 (adler32-zlib) [ 4879.191451] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 4879.520393] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 4881.311276] Key type lgssc registered [ 4882.562345] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4890.059381] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4890.095462] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4891.642153] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4896.714264] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 4901.858467] LustreError: 156367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4901.875476] LustreError: 156367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4906.992311] LustreError: 156368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4909.076990] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4912.100865] LustreError: 156367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4913.520832] LustreError: 156396:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4913.537783] LustreError: 156396:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4913.546904] LustreError: 156396:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 22, retries 0, failed: rc = -5 [ 4913.906690] Lustre: server umount lustre-MDT0000 complete [ 4932.676627] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 4943.096408] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 4987.766409] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4987.984856] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4987.999791] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4988.084803] Lustre: lustre-MDT0000: new disk, initializing [ 4988.191634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4988.241925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4993.556595] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5005.001800] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5005.200867] Lustre: 159703:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5005.245733] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5005.249137] Lustre: Skipped 1 previous similar message [ 5005.322658] Lustre: lustre-MDT0001: new disk, initializing [ 5005.447849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5005.500492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5005.513297] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5010.380862] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5021.901913] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5029.879446] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5036.823219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5037.107863] Lustre: lustre-OST0000: new disk, initializing [ 5037.111571] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5037.119506] Lustre: 161671:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5037.216931] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5044.689551] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5047.431166] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5047.449607] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5047.575789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5054.357525] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5058.499287] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5060.535993] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 02:13:43 (1788156823) [ 5353.441696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5353.452956] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5353.478059] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5354.977810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5354.978381] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5357.232283] Lustre: server umount lustre-MDT0000 complete [ 5360.111182] LustreError: 159714:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5360.128306] LustreError: 159714:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5363.093306] LustreError: 159694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157128 with bad export cookie 11058685159798335457 [ 5363.096065] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5363.121210] LustreError: 159694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5365.216544] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5365.217070] LustreError: 159711:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5365.225897] Lustre: Skipped 1 previous similar message [ 5365.230976] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5365.252516] Lustre: Skipped 2 previous similar messages [ 5368.225213] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5370.333328] Lustre: server umount lustre-MDT0001 complete [ 5382.973446] Lustre: server umount lustre-OST0000 complete [ 5394.258250] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 5410.732706] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5455.584573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5455.808522] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5455.834869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5455.934474] Lustre: lustre-MDT0000: new disk, initializing [ 5456.079626] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5456.090605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5461.847584] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5474.532592] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5474.637629] Lustre: 166362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5474.686756] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5474.693761] Lustre: Skipped 1 previous similar message [ 5474.773525] Lustre: lustre-MDT0001: new disk, initializing [ 5474.861153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5474.903062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5474.916412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5480.750627] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5492.542632] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5498.001439] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5506.554917] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5506.966962] Lustre: lustre-OST0000: new disk, initializing [ 5506.989393] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5507.001868] Lustre: 168329:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5507.136880] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5512.418861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5512.461512] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5512.725364] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5516.646381] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5529.738704] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5535.101911] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5535.550757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5540.403924] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5540.657259] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5543.395321] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5543.404786] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5543.426251] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5543.430745] Lustre: Skipped 1 previous similar message [ 5553.642589] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5553.648733] Lustre: Skipped 3 previous similar messages [ 5557.215192] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5557.473857] Lustre: server umount lustre-OST0000 complete [ 5561.835407] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5561.844987] LustreError: Skipped 1 previous similar message [ 5561.849276] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5561.862322] Lustre: Skipped 1 previous similar message [ 5574.113118] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5574.124983] Lustre: Skipped 5 previous similar messages [ 5576.159179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5576.490440] Lustre: server umount lustre-MDT0000 complete [ 5579.232718] LustreError: 166370:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5579.259636] LustreError: 166370:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5580.881768] LustreError: 166353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157346 with bad export cookie 11058685159798494294 [ 5580.892871] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5580.901609] LustreError: 166353:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5581.537437] Lustre: server umount lustre-MDT0001 complete [ 5591.634737] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:22:34 (1788157354) [ 5611.484671] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 5623.421373] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5662.558962] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5662.812241] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5662.837306] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5662.908746] Lustre: lustre-MDT0000: new disk, initializing [ 5662.996428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5663.010454] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5668.578083] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5680.328343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5680.514811] Lustre: 173090:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5680.882127] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5686.298099] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5697.106672] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5704.562126] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5713.228320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5713.880699] Lustre: lustre-OST0000: new disk, initializing [ 5713.884901] Lustre: Skipped 1 previous similar message [ 5713.899597] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5713.912946] Lustre: Skipped 2 previous similar messages [ 5713.917457] Lustre: 175069:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5719.642581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5719.652310] Lustre: Skipped 1 previous similar message [ 5719.667400] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5719.807239] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5721.809927] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5732.619891] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5745.635529] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5745.639495] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5745.651221] LustreError: Skipped 1 previous similar message [ 5745.651949] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5745.651953] Lustre: Skipped 1 previous similar message [ 5745.695050] Lustre: Skipped 2 previous similar messages [ 5754.847352] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5755.026371] Lustre: server umount lustre-OST0000 complete [ 5773.279165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5773.447854] Lustre: server umount lustre-MDT0000 complete [ 5776.362170] LustreError: 173096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5776.384590] LustreError: 173096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5778.196234] LustreError: 173081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157543 with bad export cookie 11058685159798495176 [ 5778.198596] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5778.208360] LustreError: 173081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5778.641724] Lustre: server umount lustre-MDT0001 complete [ 5788.384711] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 02:25:51 (1788157551) [ 5807.387821] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 5820.765981] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 5864.920130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5865.206900] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5865.247372] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5865.337542] Lustre: lustre-MDT0000: new disk, initializing [ 5865.393752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5865.396750] Lustre: Skipped 2 previous similar messages [ 5865.408673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5870.120801] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5882.347223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5882.492293] Lustre: 179396:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5882.860036] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5887.545375] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5897.996882] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5904.774635] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5911.824707] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5912.170530] Lustre: 181374:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5919.043073] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 5920.838777] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5920.962795] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5930.528547] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5941.731486] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5941.734069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5941.749168] Lustre: Skipped 2 previous similar messages [ 5941.758518] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5941.780120] Lustre: Skipped 9 previous similar messages [ 5946.864483] LustreError: 181354:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5947.102154] Lustre: server umount lustre-OST0000 complete [ 5966.303916] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5966.713573] Lustre: server umount lustre-MDT0000 complete [ 5970.650065] LustreError: 179387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157736 with bad export cookie 11058685159798496261 [ 5970.651175] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5970.660314] LustreError: 179387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5971.208103] Lustre: server umount lustre-MDT0001 complete [ 5984.190986] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 5987.582804] Key type lgssc unregistered [ 5987.942307] LNet: 183231:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5987.951769] LNetError: 183231:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5987.972676] LNet: Removed LNI 192.168.201.109@tcp [ 5988.728349] Key type .llcrypt unregistered [ 5988.730972] Key type ._llcrypt unregistered [ 6016.427732] Key type ._llcrypt registered [ 6016.431794] Key type .llcrypt registered [ 6016.591096] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6017.665618] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6017.706932] alg: No test for adler32 (adler32-zlib) [ 6018.845524] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6019.149587] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6020.879201] Key type lgssc registered [ 6022.027955] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6031.540085] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6031.568060] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6033.290315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6038.323733] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6042.613041] LustreError: 184655:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6042.645054] LustreError: 184655:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6047.716091] LustreError: 184654:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6048.163179] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6048.586712] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6053.775088] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6062.661454] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6069.071980] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6076.241850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6076.759584] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6077.834904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6084.345048] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6095.252198] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6118.880417] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6118.893469] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6118.914042] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6119.910189] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6119.918108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6119.920915] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6119.938877] Lustre: Skipped 1 previous similar message [ 6121.846093] Lustre: server umount lustre-OST0000 complete [ 6130.147291] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6130.151585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6130.159840] Lustre: Skipped 1 previous similar message [ 6130.190735] Lustre: Skipped 1 previous similar message [ 6133.569993] Lustre: server umount lustre-MDT0000 complete [ 6135.265229] LustreError: 184650:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6135.306635] LustreError: 184650:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6138.176588] LustreError: 184635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788157903 with bad export cookie 18353164533779445752 [ 6138.182571] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6138.191372] LustreError: 184635:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6138.884886] Lustre: server umount lustre-MDT0001 complete [ 6152.442372] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 6155.568623] Key type lgssc unregistered [ 6156.160152] LNet: 188209:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6156.166406] LNetError: 188209:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6156.188669] LNet: Removed LNI 192.168.201.109@tcp [ 6156.971276] Key type .llcrypt unregistered [ 6156.979468] Key type ._llcrypt unregistered [ 6185.604422] Key type ._llcrypt registered [ 6185.607553] Key type .llcrypt registered [ 6185.705662] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6187.283280] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6187.319923] alg: No test for adler32 (adler32-zlib) [ 6188.482537] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 6188.794816] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 6190.519605] Key type lgssc registered [ 6191.796831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6200.114884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6200.146537] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6201.885031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6206.898959] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6212.069929] LustreError: 189625:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6212.090157] LustreError: 189625:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6215.567072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6215.934637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6220.851324] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6228.979329] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6234.979460] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6241.999572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6242.374086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6246.465372] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6248.569047] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6256.545744] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6266.852352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6266.860827] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6266.883962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6269.320745] Lustre: lustre-OST0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 6270.949929] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6270.952320] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6270.956045] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6274.436408] Lustre: lustre-OST0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 6274.445374] Lustre: Skipped 1 previous similar message [ 6279.532394] Lustre: lustre-OST0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 6279.549105] Lustre: Skipped 2 previous similar messages [ 6281.194185] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6281.484106] Lustre: server umount lustre-OST0000 complete [ 6289.654583] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9e2e0517c800.ost_server_uuid 50 [ 6291.528774] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6308.322116] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6308.328260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6308.339604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6308.350103] Lustre: Skipped 2 previous similar messages [ 6311.393094] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.366513] Lustre: server umount lustre-MDT0000 complete [ 6316.518053] LustreError: 189621:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.540855] LustreError: 189621:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 6317.577147] LustreError: 189605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158083 with bad export cookie 7727207329611514039 [ 6317.584880] LustreError: 189605:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6317.588294] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6318.003860] Lustre: server umount lustre-MDT0001 complete [ 6328.350508] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 02:34:51 (1788158091) [ 6336.001203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.584857] LustreError: 193546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6336.664488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6341.064824] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6342.113707] LustreError: 193547:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6347.235648] LustreError: 193546:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6351.391063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6351.779457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6358.120568] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6367.999330] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6375.187565] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6382.454631] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6382.753772] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6388.998638] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6397.421993] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6406.634439] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6406.916364] Lustre: lustre-OST0001: new disk, initializing [ 6406.919925] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6406.933534] Lustre: 196388:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6407.076731] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6408.168859] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6413.980714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6414.013583] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6414.197633] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6414.808118] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6426.810270] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6429.674708] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6429.703422] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6429.710454] Lustre: Skipped 3 previous similar messages [ 6435.691951] Lustre: server umount lustre-OST0000 complete [ 6437.749783] LustreError: 195371:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6443.543686] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6443.977446] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6449.068466] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6449.304649] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6455.298665] LustreError: 196705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6455.347106] LustreError: 196705:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 10 previous similar messages [ 6465.519051] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6465.523451] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6465.543417] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6465.550533] Lustre: Skipped 2 previous similar messages [ 6465.581447] Lustre: Skipped 4 previous similar messages [ 6467.634105] Lustre: server umount lustre-OST0001 complete [ 6474.731752] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6474.736402] LustreError: Skipped 1 previous similar message [ 6479.223809] Lustre: server umount lustre-MDT0000 complete [ 6483.414753] LustreError: 194284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158249 with bad export cookie 7727207329611515103 [ 6483.424752] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6483.431802] LustreError: 194284:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6483.830883] Lustre: server umount lustre-MDT0001 complete [ 6510.587402] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 02:37:53 (1788158273) [ 6517.582730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6517.905060] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6517.935608] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6518.069158] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6523.491160] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6533.141952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6533.280728] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6533.349911] Lustre: 199829:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6538.503675] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6546.706717] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6552.874958] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6559.835731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6560.076245] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6567.414297] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6568.109886] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6576.182475] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6583.500215] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6583.602658] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6583.824124] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6583.833482] Lustre: Skipped 2 previous similar messages [ 6591.199947] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6592.729709] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6601.567122] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6607.273943] Lustre: lustre-OST0000: Not available for connect from 192.168.201.9@tcp (stopping) [ 6607.286497] Lustre: Skipped 3 previous similar messages [ 6608.358916] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6608.384996] Lustre: Skipped 3 previous similar messages [ 6611.267714] Lustre: server umount lustre-OST0000 complete [ 6612.346738] LustreError: 201677:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6612.364365] LustreError: 201677:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 6628.835470] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6634.446947] Lustre: server umount lustre-OST0001 complete [ 6645.358038] Lustre: server umount lustre-MDT0000 complete [ 6649.095392] LustreError: 199820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158414 with bad export cookie 7727207329611516517 [ 6649.097415] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6649.105116] LustreError: 199820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6649.401064] Lustre: server umount lustre-MDT0001 complete [ 6677.240839] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 02:40:40 (1788158440) [ 6695.676700] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 6707.325592] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 6744.774657] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6744.981274] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6744.999643] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6745.053892] Lustre: lustre-MDT0000: new disk, initializing [ 6745.100453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6745.113886] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6749.260284] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6758.971442] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6759.068339] Lustre: 207929:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6759.108803] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6759.111567] Lustre: Skipped 1 previous similar message [ 6759.161933] Lustre: lustre-MDT0001: new disk, initializing [ 6759.254828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6759.265862] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6763.722589] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6774.067588] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6780.204812] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6787.008358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6787.183880] Lustre: lustre-OST0000: new disk, initializing [ 6787.186634] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6787.191392] Lustre: 209897:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6793.946655] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6796.856341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6796.878807] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6797.010213] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6804.630645] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6810.266956] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6810.498479] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6816.478503] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6816.860771] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6822.896468] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6822.898493] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6822.906505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6822.906513] Lustre: Skipped 10 previous similar messages [ 6822.913304] Lustre: Skipped 5 previous similar messages [ 6822.953701] LustreError: Skipped 2 previous similar messages [ 6825.477894] Lustre: server umount lustre-OST0000 complete [ 6838.066388] Lustre: server umount lustre-MDT0000 complete [ 6838.242382] LustreError: 208929:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6838.256780] LustreError: 208929:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 6842.115891] LustreError: 207920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158607 with bad export cookie 7727207329611518295 [ 6842.118418] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6842.129083] LustreError: 207920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6842.608398] Lustre: server umount lustre-MDT0001 complete [ 6852.833522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6858.549140] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6867.484241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6872.726864] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6881.294061] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6886.647275] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6895.125243] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6902.494091] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6909.727833] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6914.949618] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6915.206626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6919.549828] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6919.742957] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6923.235134] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6923.237547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6923.257542] Lustre: Skipped 3 previous similar messages [ 6927.828309] Lustre: server umount lustre-OST0000 complete [ 6934.615490] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6938.985771] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6969.528446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6977.785705] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 6986.324864] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6987.498701] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6987.516119] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6987.519051] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6987.536020] Lustre: Skipped 1 previous similar message [ 6988.028299] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6992.869615] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6992.882114] LustreError: Skipped 1 previous similar message [ 6999.095759] Lustre: server umount lustre-OST0000 complete [ 7009.791874] Lustre: server umount lustre-MDT0000 complete [ 7013.352585] LustreError: 212843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7013.371420] LustreError: 212843:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 7013.858240] LustreError: 212096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788158779 with bad export cookie 7727207329611519184 [ 7013.878151] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7014.309620] Lustre: server umount lustre-MDT0001 complete [ 7026.942336] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 02:46:29 (1788158789) [ 7036.775760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7037.369188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7037.373312] Lustre: Skipped 6 previous similar messages [ 7041.746842] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7051.075830] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7056.481726] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7065.361507] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7072.092838] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7079.535706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7086.418229] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7094.948035] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7100.141914] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7100.335711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7105.854289] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7106.085598] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7113.518486] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7113.850786] Lustre: lustre-OST0001: new disk, initializing [ 7113.854091] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7113.860995] Lustre: 221457:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7120.583036] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7120.610393] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7120.835875] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7122.526279] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7133.702757] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7138.827973] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7139.160559] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7145.011391] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7145.284492] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7151.600194] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7151.600294] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7151.608086] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7151.608095] Lustre: Skipped 12 previous similar messages [ 7151.608695] LustreError: Skipped 2 previous similar messages [ 7151.634876] Lustre: Skipped 5 previous similar messages [ 7153.856482] Lustre: server umount lustre-OST0001 complete [ 7160.420511] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7160.644044] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7165.213856] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7165.506100] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7193.704874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7199.204877] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7199.217461] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7199.228556] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7199.235400] Lustre: Skipped 1 previous similar message [ 7199.446560] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7206.842466] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7208.503821] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7225.311176] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7225.454280] Lustre: server umount lustre-OST0001 complete [ 7248.685652] Lustre: server umount lustre-OST0000 complete [ 7258.439980] Lustre: server umount lustre-MDT0000 complete [ 7261.957037] LustreError: 218067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159027 with bad export cookie 7727207329611520150 [ 7261.965354] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7261.976249] LustreError: 218067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 7268.555942] Lustre: server umount lustre-MDT0001 complete [ 7291.604405] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 02:50:54 (1788159054) [ 7298.009179] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7298.274483] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7298.313229] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7302.626063] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7311.369975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7311.490307] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7311.532625] Lustre: 226707:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7316.637585] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7325.049063] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7331.470179] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7337.835272] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7338.009803] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7343.754159] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7352.069978] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7361.855675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7361.900190] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7368.772123] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7377.571675] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7379.784445] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7384.945410] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7385.237663] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7390.614875] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7390.837599] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7397.866126] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9e2e080d4800.ost_server_uuid 50 [ 7400.189717] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9e2e080d4800.ost_server_uuid in IDLE state after 0 sec [ 7400.975797] Lustre: Permanently deactivating lustre-OST0001 [ 7400.989975] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7405.438028] Lustre: Permanently reactivating lustre-OST0001 [ 7405.452127] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 7405.456400] Lustre: Skipped 2 previous similar messages [ 7411.179096] LustreError: lustre-OST0001-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 7411.193711] LustreError: Skipped 2 previous similar messages [ 7411.200082] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7411.219511] Lustre: Skipped 6 previous similar messages [ 7415.723364] Lustre: server umount lustre-OST0001 complete [ 7417.318369] LustreError: 228552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7417.338539] LustreError: 228552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 37 previous similar messages [ 7425.580683] Lustre: server umount lustre-OST0000 complete [ 7435.657829] Lustre: server umount lustre-MDT0000 complete [ 7439.535947] LustreError: 228567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159205 with bad export cookie 7727207329611521438 [ 7439.546525] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7440.259050] Lustre: server umount lustre-MDT0001 complete [ 7465.969192] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 02:53:49 (1788159229) [ 7474.866286] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7475.284227] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7475.338464] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7475.347463] Lustre: Skipped 2 previous similar messages [ 7480.233487] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7488.907312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7489.031728] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7489.069803] Lustre: 233402:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7493.000531] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7501.020624] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7506.774549] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7513.715423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7514.024498] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7514.052986] Lustre: Permanently deactivating lustre-OST0000 [ 7514.071467] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7520.278399] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7520.655327] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7529.584508] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7538.722185] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7538.771259] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7544.729538] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7548.732868] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7558.009657] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7560.339461] Lustre: Permanently reactivating lustre-OST0000 [ 7560.359929] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7560.365449] Lustre: Skipped 2 previous similar messages [ 7566.893949] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7566.894608] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 46 seconds. I think it's dead, and I am evicting it. exp 00000000a95e06fc, cur 1788159332 deadline 1788159316 last 1788159286 [ 7566.926450] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7566.947715] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7567.028652] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7579.254753] Lustre: server umount lustre-OST0001 complete [ 7589.444695] Lustre: server umount lustre-OST0000 complete [ 7599.445589] Lustre: server umount lustre-MDT0000 complete [ 7603.727115] LustreError: 233393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159369 with bad export cookie 7727207329611523097 [ 7603.727789] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7603.740875] LustreError: 233393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7604.133220] Lustre: server umount lustre-MDT0001 complete [ 7619.531343] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 7623.722661] Key type lgssc unregistered [ 7624.354543] LNet: 238281:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7624.372522] LNetError: 238281:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7624.394042] LNet: Removed LNI 192.168.201.109@tcp [ 7625.638733] Key type .llcrypt unregistered [ 7625.642581] Key type ._llcrypt unregistered [ 7657.878139] Key type ._llcrypt registered [ 7657.883640] Key type .llcrypt registered [ 7657.980942] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7659.324662] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7659.370342] alg: No test for adler32 (adler32-zlib) [ 7660.512940] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 7660.777855] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 7662.439298] Key type lgssc registered [ 7663.994195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7676.183107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7676.193328] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7677.872726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7683.002079] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7688.177258] LustreError: 239752:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7688.189823] LustreError: 239752:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7693.302933] LustreError: 239753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7693.482186] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7693.638669] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7693.681745] Lustre: 239742:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7693.696711] Lustre: Permanently deactivating lustre-MDT0001 [ 7693.726120] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7694.166417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7696.863531] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7696.879617] LustreError: 239783:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7696.903099] LustreError: 239783:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7696.928621] LustreError: 239783:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 1, failed: rc = -108 [ 7698.805478] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7706.603279] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7712.440160] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7719.982677] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7720.379612] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7722.419158] LustreError: 241593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7722.442117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7726.355887] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7727.587467] LustreError: 241593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7732.719395] LustreError: 241593:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7734.890882] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7742.871297] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7743.212717] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7748.599049] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7748.604064] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7749.089604] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7756.845925] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7761.126770] Lustre: Permanently reactivating lustre-MDT0001 [ 7761.170369] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7761.174130] Lustre: Skipped 2 previous similar messages [ 7764.870879] Lustre: lustre-MDT0001: Client f8cdb28b-c5ba-436b-a357-a603f822005a (at 192.168.201.9@tcp) reconnecting [ 7771.183788] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7772.829953] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7779.391709] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7779.757925] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7785.280861] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7785.614409] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7789.132456] Lustre: Permanently deactivating lustre-MDT0001 [ 7789.172768] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7789.180764] Lustre: Skipped 2 previous similar messages [ 7797.890693] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7807.989266] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7808.004698] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7808.027967] Lustre: Skipped 2 previous similar messages [ 7808.043849] Lustre: Skipped 3 previous similar messages [ 7813.090413] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7813.103745] Lustre: Skipped 3 previous similar messages [ 7814.161394] Lustre: server umount lustre-MDT0000 complete [ 7818.211369] LustreError: 239753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7818.224646] LustreError: 239753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 7818.736489] LustreError: 239733:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159584 with bad export cookie 11941707364350319198 [ 7818.752295] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7818.945986] Lustre: server umount lustre-MDT0001 complete [ 7828.739648] Lustre: server umount lustre-OST0000 complete [ 7841.091305] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 03:00:04 (1788159604) [ 7852.895204] Lustre: server umount lustre-OST0001 complete [ 7861.985667] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 7873.664791] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 7915.838616] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.314044] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7916.328407] Lustre: Skipped 2 previous similar messages [ 7916.384742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7916.542078] Lustre: lustre-MDT0000: new disk, initializing [ 7916.720698] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7916.787826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7922.684661] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7935.244258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7935.429662] Lustre: 248038:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7935.496705] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7935.504407] Lustre: Skipped 1 previous similar message [ 7935.613090] Lustre: lustre-MDT0001: new disk, initializing [ 7935.733259] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7935.775699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7935.792254] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7941.138513] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7953.488063] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7960.834757] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7969.282741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7969.761865] Lustre: lustre-OST0000: new disk, initializing [ 7969.787435] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7969.807541] Lustre: 250005:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7970.027961] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7977.109333] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7977.133531] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7977.368861] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7979.688065] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 7993.460657] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7996.867550] LustreError: 248043:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8006.565406] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8006.693670] Lustre: lustre-OST0001: new disk, initializing [ 8006.698053] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8006.701993] Lustre: 251120:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8006.800224] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8006.935198] LustreError: 248043:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8006.971528] LustreError: 250470:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8013.406715] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8016.008436] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8016.020841] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8016.158359] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8017.005031] LustreError: 250470:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8026.211180] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8027.921576] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8031.713160] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8031.718109] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8031.723657] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8031.732042] Lustre: Skipped 2 previous similar messages [ 8031.740182] Lustre: Skipped 1 previous similar message [ 8036.357887] Lustre: server umount lustre-OST0001 complete [ 8036.848851] LustreError: 251585:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8036.867688] LustreError: 251585:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8054.245709] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8054.251037] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8054.261827] LustreError: Skipped 2 previous similar messages [ 8054.264513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8054.281534] Lustre: Skipped 1 previous similar message [ 8060.326217] Lustre: server umount lustre-OST0000 complete [ 8067.554247] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8067.565505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8067.571819] Lustre: Skipped 1 previous similar message [ 8067.582806] Lustre: Skipped 4 previous similar messages [ 8070.899437] Lustre: server umount lustre-MDT0000 complete [ 8072.674622] LustreError: 248049:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8072.695508] LustreError: 248049:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 9 previous similar messages [ 8074.662421] LustreError: 248028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788159840 with bad export cookie 11941707364350321795 [ 8074.666544] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8074.681615] LustreError: 248028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 8081.409457] Lustre: server umount lustre-MDT0001 complete [ 8093.402120] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8096.569386] Key type lgssc unregistered [ 8096.908947] LNet: 253155:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8096.922850] LNetError: 253155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8096.940662] LNet: Removed LNI 192.168.201.109@tcp [ 8098.082058] Key type .llcrypt unregistered [ 8098.084162] Key type ._llcrypt unregistered [ 8141.905499] Key type ._llcrypt registered [ 8141.907415] Key type .llcrypt registered [ 8141.990854] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8143.413345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8143.512025] alg: No test for adler32 (adler32-zlib) [ 8144.539260] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8144.755056] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8146.408592] Key type lgssc registered [ 8147.325147] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8154.055070] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8154.083573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8155.458508] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8155.475381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8155.617433] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8159.982969] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8169.109227] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8169.188096] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8169.209284] Lustre: 255364:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8169.427164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8173.669374] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8182.454718] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8189.315551] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8196.963421] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8197.099136] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8197.259653] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8205.483242] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8206.976808] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8213.288899] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8268.263583] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8268.273416] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8268.296353] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8268.784693] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8268.794167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8268.828936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8273.817350] Lustre: server umount lustre-OST0000 complete [ 8278.329038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8291.186828] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8291.579271] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8298.279321] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8306.879599] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8308.962973] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8308.983937] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8308.984888] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8309.021384] Lustre: Skipped 1 previous similar message [ 8350.186472] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8350.189052] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8350.198128] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8350.207528] Lustre: Skipped 1 previous similar message [ 8350.212366] LustreError: Skipped 1 previous similar message [ 8353.295915] Lustre: server umount lustre-OST0000 complete [ 8358.883279] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8358.888067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8358.906227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8358.909694] Lustre: Skipped 1 previous similar message [ 8364.118634] Lustre: server umount lustre-MDT0000 complete [ 8368.507339] LustreError: 256107:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160134 with bad export cookie 5170953813265397135 [ 8368.522686] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8368.904226] Lustre: server umount lustre-MDT0001 complete [ 8380.560741] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8382.874348] Key type lgssc unregistered [ 8383.140952] LNet: 260792:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8383.156788] LNetError: 260792:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8383.185558] LNet: Removed LNI 192.168.201.109@tcp [ 8384.212151] Key type .llcrypt unregistered [ 8384.214434] Key type ._llcrypt unregistered [ 8412.166688] Key type ._llcrypt registered [ 8412.169831] Key type .llcrypt registered [ 8412.339962] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8413.545590] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8413.734834] alg: No test for adler32 (adler32-zlib) [ 8414.735824] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8415.137282] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8416.833062] Key type lgssc registered [ 8418.186025] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8427.160195] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8427.189983] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8428.792202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8433.524252] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8439.276622] LustreError: 262211:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8439.311530] LustreError: 262211:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8443.213267] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8443.904534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8449.185258] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8458.545583] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8464.580277] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8471.613404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8472.039507] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8474.102274] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8479.525224] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8488.760038] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8519.136425] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8519.145192] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8519.159918] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8519.183715] Lustre: Skipped 1 previous similar message [ 8519.210638] Lustre: Skipped 1 previous similar message [ 8524.283167] LustreError: 264034:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8524.587370] Lustre: server umount lustre-OST0000 complete [ 8530.912529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8530.931532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8530.954417] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8534.497987] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8534.499322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8534.530442] Lustre: Skipped 1 previous similar message [ 8537.217910] Lustre: server umount lustre-MDT0000 complete [ 8539.619107] LustreError: 262205:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8539.653396] LustreError: 262205:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8542.225044] LustreError: 262191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160307 with bad export cookie 14986213458071892907 [ 8542.235373] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8542.246569] LustreError: 262191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8542.907057] Lustre: server umount lustre-MDT0001 complete [ 8556.139507] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8559.674689] Key type lgssc unregistered [ 8560.061734] LNet: 266519:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8560.081036] LNetError: 266519:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8560.137198] LNet: Removed LNI 192.168.201.109@tcp [ 8561.380750] Key type .llcrypt unregistered [ 8561.388417] Key type ._llcrypt unregistered [ 8582.308798] Key type ._llcrypt registered [ 8582.310613] Key type .llcrypt registered [ 8582.433487] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8583.941831] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8584.072854] alg: No test for adler32 (adler32-zlib) [ 8585.206922] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8585.499070] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8587.200369] Key type lgssc registered [ 8589.126935] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8597.823907] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8597.849110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8599.362746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8605.339964] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8608.738389] LustreError: 267753:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8608.774859] LustreError: 267753:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8613.858689] LustreError: 267754:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8615.971777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8616.494237] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8622.025567] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8634.031974] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8642.876404] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8650.957045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8651.631979] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8659.162149] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8668.114973] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8669.036664] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8684.520585] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8684.526856] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8684.541238] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8684.547854] Lustre: Skipped 1 previous similar message [ 8688.505703] Lustre: server umount lustre-OST0000 complete [ 8693.216941] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8693.227631] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8693.251250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8693.256939] Lustre: Skipped 1 previous similar message [ 8694.752879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8694.754506] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8698.944303] Lustre: server umount lustre-MDT0000 complete [ 8699.872386] LustreError: 269846:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8699.887157] LustreError: 269846:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8703.355712] LustreError: 269598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160468 with bad export cookie 4600775215695113355 [ 8703.365554] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8703.374498] LustreError: 269598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8703.767158] Lustre: server umount lustre-MDT0001 complete [ 8720.394694] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8724.757800] Key type lgssc unregistered [ 8725.034995] LNet: 271451:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8725.048388] LNetError: 271451:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8725.062645] LNet: Removed LNI 192.168.201.109@tcp [ 8726.308095] Key type .llcrypt unregistered [ 8726.309801] Key type ._llcrypt unregistered [ 8754.497123] Key type ._llcrypt registered [ 8754.499849] Key type .llcrypt registered [ 8754.612271] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8755.602188] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8755.634668] alg: No test for adler32 (adler32-zlib) [ 8756.937875] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8757.294104] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8758.961796] Key type lgssc registered [ 8760.338903] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8770.306546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8770.331972] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8771.968048] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8777.342295] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8782.315633] LustreError: 272868:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8782.364888] LustreError: 272868:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8787.430798] LustreError: 272869:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8788.727628] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8789.089772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8793.784552] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8802.260709] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8809.374440] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8817.073300] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8817.509248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8825.458287] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8835.063869] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8842.112417] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8869.856377] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8869.862174] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8869.882751] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8872.935119] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8872.947716] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8872.951482] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8872.978166] Lustre: Skipped 1 previous similar message [ 8874.626672] Lustre: server umount lustre-OST0000 complete [ 8881.123047] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8881.128214] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8881.140694] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8883.171929] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8883.177088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8883.201979] Lustre: Skipped 1 previous similar message [ 8885.439580] Lustre: server umount lustre-MDT0000 complete [ 8888.287975] LustreError: 273598:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8888.295881] LustreError: 273598:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 8888.698714] LustreError: 272849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160654 with bad export cookie 9328845202233393917 [ 8888.716685] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8889.128945] Lustre: server umount lustre-MDT0001 complete [ 8901.320661] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 8903.831563] Key type lgssc unregistered [ 8904.157923] LNet: 277200:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8904.165559] LNetError: 277200:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8905.192117] LNet: Removed LNI 192.168.201.109@tcp [ 8906.487838] Key type .llcrypt unregistered [ 8906.490391] Key type ._llcrypt unregistered [ 8927.952134] Key type ._llcrypt registered [ 8927.955430] Key type .llcrypt registered [ 8928.149085] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 8929.956425] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8929.981854] alg: No test for adler32 (adler32-zlib) [ 8931.189273] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 8931.536819] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 8933.279160] Key type lgssc registered [ 8934.580083] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8943.373523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8943.388615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8944.908911] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8949.419880] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8955.361120] LustreError: 278445:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8955.390613] LustreError: 278445:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8959.387144] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8959.938281] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8965.748612] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 8974.873300] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8982.346588] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8990.418905] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8990.850484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8999.971636] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9009.619119] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9015.421152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9031.145917] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9031.149664] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9031.154473] LustreError: Skipped 1 previous similar message [ 9031.155401] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9031.200142] Lustre: Skipped 1 previous similar message [ 9033.841595] Lustre: server umount lustre-OST0000 complete [ 9041.380522] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9041.408164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9041.418812] Lustre: Skipped 2 previous similar messages [ 9044.446597] Lustre: server umount lustre-MDT0000 complete [ 9046.499207] LustreError: 278440:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9046.524702] LustreError: 278440:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9048.937488] LustreError: 278410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788160814 with bad export cookie 8180709007214420039 [ 9048.964315] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9049.474437] Lustre: server umount lustre-MDT0001 complete [ 9064.600722] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9069.005245] Key type lgssc unregistered [ 9069.505616] LNet: 282149:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9069.512593] LNetError: 282149:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9069.527560] LNet: Removed LNI 192.168.201.109@tcp [ 9070.676781] Key type .llcrypt unregistered [ 9070.678811] Key type ._llcrypt unregistered [ 9086.453776] Key type ._llcrypt registered [ 9086.455016] Key type .llcrypt registered [ 9086.709905] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9105.720718] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 9123.911681] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9125.312199] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9125.609436] alg: No test for adler32 (adler32-zlib) [ 9126.733660] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9127.147988] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 9128.855193] Key type lgssc registered [ 9130.009989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9168.067117] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9168.121892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9169.504315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9169.560754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9169.651321] Lustre: lustre-MDT0000: new disk, initializing [ 9169.711202] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9169.723981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9175.855769] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9187.755988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9187.863279] Lustre: 285712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9187.916508] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9187.922755] Lustre: Skipped 1 previous similar message [ 9187.993424] Lustre: lustre-MDT0001: new disk, initializing [ 9188.069301] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9188.106552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9188.117757] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9193.253605] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9203.266285] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9210.200416] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9218.387039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9218.872342] Lustre: lustre-OST0000: new disk, initializing [ 9218.875309] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9218.881155] Lustre: 287682:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9218.970214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9226.306792] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9229.414037] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9229.426099] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9229.540223] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9236.097869] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9241.125800] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9241.352545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9246.990939] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9247.211241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9250.274183] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9250.275264] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9250.291340] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9250.321726] Lustre: Skipped 1 previous similar message [ 9250.361683] Lustre: Skipped 1 previous similar message [ 9255.540263] Lustre: server umount lustre-OST0000 complete [ 9260.001337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9260.007823] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9260.017292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9265.633726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9265.645559] Lustre: Skipped 3 previous similar messages [ 9265.822881] Lustre: server umount lustre-MDT0000 complete [ 9270.019768] LustreError: 285703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161035 with bad export cookie 4357142935992808647 [ 9270.029508] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9270.031628] LustreError: 285703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9270.439244] Lustre: server umount lustre-MDT0001 complete [ 9279.148621] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 03:24:02 (1788161042) [ 9286.063666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9286.361254] LustreError: 290093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9286.377080] LustreError: 290093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9286.431137] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9290.759121] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9291.745969] LustreError: 290094:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9296.870537] LustreError: 290093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9298.941859] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9299.325043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9304.359993] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9312.553657] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9318.651262] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9325.987848] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9326.304033] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9332.053561] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9339.585680] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9354.210465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9354.217526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9354.225258] Lustre: Skipped 1 previous similar message [ 9354.232786] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9354.260074] Lustre: Skipped 1 previous similar message [ 9359.214406] Lustre: server umount lustre-OST0000 complete [ 9364.448991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9364.474487] Lustre: Skipped 1 previous similar message [ 9364.479073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9364.492442] Lustre: Skipped 1 previous similar message [ 9370.265469] Lustre: server umount lustre-MDT0000 complete [ 9373.665168] LustreError: 290093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9373.687349] LustreError: 290093:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9374.093847] LustreError: 290074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161139 with bad export cookie 4357142935992809522 [ 9374.106160] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9374.110751] LustreError: 290074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9374.664084] Lustre: server umount lustre-MDT0001 complete [ 9388.695244] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9391.968047] Key type lgssc unregistered [ 9392.398595] LNet: 293650:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9392.402798] LNetError: 293650:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9392.418931] LNet: Removed LNI 192.168.201.109@tcp [ 9393.612195] Key type .llcrypt unregistered [ 9393.614557] Key type ._llcrypt unregistered [ 9424.286709] Key type ._llcrypt registered [ 9424.289216] Key type .llcrypt registered [ 9424.389461] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 9443.069696] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9443.992659] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9444.018506] alg: No test for adler32 (adler32-zlib) [ 9445.151873] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9445.539426] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [ 9447.279150] Key type lgssc registered [ 9448.734848] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9510.112918] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9510.146884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9511.487599] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9511.521829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9511.621590] Lustre: lustre-MDT0000: new disk, initializing [ 9511.746650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9511.786959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9516.939962] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9527.332361] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9527.453555] Lustre: 297482:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9527.501984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9527.505497] Lustre: Skipped 1 previous similar message [ 9527.594918] Lustre: lustre-MDT0001: new disk, initializing [ 9527.724995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9527.764191] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9527.785243] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9534.004473] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9547.285608] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9555.784978] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9565.056935] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9565.507738] Lustre: lustre-OST2710: new disk, initializing [ 9565.512214] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9565.525907] Lustre: 299451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9565.692089] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9572.753591] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9574.038117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9574.053621] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9574.275052] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9582.918720] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9590.052821] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9590.216199] Lustre: lustre-OST03e8: new disk, initializing [ 9590.219909] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9590.224912] Lustre: 300518:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9590.300202] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9596.797662] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9598.031290] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9598.036559] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9598.092189] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9603.526690] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9612.397346] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9612.615976] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9617.308135] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9617.562352] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9623.919479] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9624.359643] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9629.922724] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9630.147386] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9635.814167] Lustre: server umount lustre-MDT0000 complete [ 9639.137947] LustreError: 297472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161404 with bad export cookie 13735067426677647244 [ 9639.150506] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9639.155374] LustreError: 297472:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9639.400136] Lustre: lustre-MDT0000-lwp-OST03e8: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9639.410283] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9639.411159] Lustre: Skipped 3 previous similar messages [ 9639.428056] Lustre: Skipped 1 previous similar message [ 9641.986520] Lustre: server umount lustre-MDT0001 complete [ 9646.926660] Lustre: server umount lustre-OST2710 complete [ 9651.849872] Lustre: server umount lustre-OST03e8 complete [ 9676.672150] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [ 9689.326255] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9730.941825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9731.401073] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9731.479579] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9731.570367] Lustre: lustre-MDT0000: new disk, initializing [ 9731.701428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9731.722969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9738.078508] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9749.992552] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.148659] Lustre: 305769:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9750.280213] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9750.286721] Lustre: Skipped 1 previous similar message [ 9750.400805] Lustre: lustre-MDT0001: new disk, initializing [ 9750.568336] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9750.603535] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9750.618145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9754.644853] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9765.070639] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9772.112442] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9779.554972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9779.881297] Lustre: lustre-OST0000: new disk, initializing [ 9779.886850] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9779.903521] Lustre: 307738:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9779.980577] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9786.538768] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9790.066413] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9790.075988] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9790.246307] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9797.844402] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9803.715650] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9803.978938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9808.689951] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9808.923487] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9816.035500] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9816.044820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9816.046343] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9817.313804] Lustre: server umount lustre-OST0000 complete [ 9822.177305] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9822.206862] LustreError: Skipped 1 previous similar message [ 9822.215049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9822.238195] Lustre: Skipped 1 previous similar message [ 9822.249756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9822.258160] Lustre: Skipped 1 previous similar message [ 9826.283735] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9826.286647] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9826.312160] Lustre: Skipped 1 previous similar message [ 9828.152091] Lustre: server umount lustre-MDT0000 complete [ 9831.392049] LustreError: 305780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9831.428987] LustreError: 305780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 9832.238217] LustreError: 305759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161597 with bad export cookie 13735067426677648924 [ 9832.242113] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9832.254724] LustreError: 305759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9832.688582] Lustre: server umount lustre-MDT0001 complete [ 9845.515887] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 03:33:28 (1788161608) [ 9848.466662] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9851.305697] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 03:33:33 (1788161613) [ 9856.431487] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9894.760807] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9894.933571] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9894.955849] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9895.115518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9901.285055] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9913.609704] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9913.807838] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9913.857717] Lustre: 311597:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9920.136193] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [ 9943.226111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9943.608972] Lustre: Denying initial registration attempt from nid 192.168.201.109@tcp, specified as failover [ 9943.616779] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9943.625436] LustreError: 313053:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -99 [ 9943.630830] LustreError: 313053:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [ 9943.635530] LustreError: 313053:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9943.715186] Lustre: server umount lustre-OST0000 complete [ 9943.719615] LustreError: 313053:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9952.737225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9952.773171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9952.778971] Lustre: Skipped 1 previous similar message [ 9957.560249] Lustre: server umount lustre-MDT0000 complete [ 9957.867301] LustreError: 312695:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9957.893621] LustreError: 312695:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9962.444851] LustreError: 311590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161728 with bad export cookie 13735067426677649806 [ 9962.450457] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9962.455784] LustreError: 311590:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9962.838553] Lustre: server umount lustre-MDT0001 complete [ 9976.182802] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [ 9980.602240] Key type lgssc unregistered [ 9980.927718] LNet: 314175:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9980.930995] LNetError: 314175:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9980.948523] LNet: Removed LNI 192.168.201.109@tcp [ 9981.901446] Key type .llcrypt unregistered [ 9981.903428] Key type ._llcrypt unregistered [ 9996.187887] Key type ._llcrypt registered [ 9996.190320] Key type .llcrypt registered [ 9996.301800] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [ 9997.789507] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9997.931303] alg: No test for adler32 (adler32-zlib) [ 9999.040412] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [ 9999.357839] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [10001.063326] Key type lgssc registered [10002.367068] Lustre: Echo OBD driver; http://www.lustre.org/ [10050.678344] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [10060.841399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10060.878520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10062.175887] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10062.202324] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10062.468241] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10067.848723] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10079.392447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10079.471203] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10079.497471] Lustre: 316941:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10079.655052] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10085.348466] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10095.767754] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10101.866762] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10122.673288] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10123.021392] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10123.276151] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10130.437631] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10139.859492] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10142.701528] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10142.704783] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10142.724549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10147.300972] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10147.305857] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10147.323775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10148.980272] Lustre: server umount lustre-OST0000 complete [10156.512862] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10156.519673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10156.539888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10156.541720] Lustre: Skipped 1 previous similar message [10160.658555] Lustre: server umount lustre-MDT0000 complete [10165.191081] LustreError: 319764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788161930 with bad export cookie 15841645178475991120 [10165.196913] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10165.211681] LustreError: 319764:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10165.794567] Lustre: server umount lustre-MDT0001 complete [10181.242182] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [10185.278938] Key type lgssc unregistered [10185.608925] LNet: 320517:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10185.617178] LNetError: 320517:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10185.640949] LNet: Removed LNI 192.168.201.109@tcp [10186.435602] Key type .llcrypt unregistered [10186.439872] Key type ._llcrypt unregistered [10213.603407] Key type ._llcrypt registered [10213.607094] Key type .llcrypt registered [10213.698610] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [10213.973303] Key type .llcrypt unregistered [10213.975035] Key type ._llcrypt unregistered [10233.135395] Key type ._llcrypt registered [10233.136777] Key type .llcrypt registered [10233.244663] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [10235.139784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10235.179456] alg: No test for adler32 (adler32-zlib) [10236.351311] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10236.713379] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [10238.376095] Key type lgssc registered [10239.578350] Lustre: Echo OBD driver; http://www.lustre.org/ [10274.479952] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [10275.579973] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [10291.915931] loop0: detected capacity change from 0 to 204800000 [10292.189711] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10292.202635] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10292.217628] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10311.377354] loop0: detected capacity change from 0 to 204800000 [10311.539626] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10311.548274] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10311.552491] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10331.666977] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10331.735425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10333.111565] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10333.126867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10333.333237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10338.280167] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10348.093247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10348.237825] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10348.334111] Lustre: 324446:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10348.699503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10353.404189] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10362.077759] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10369.312383] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10376.153294] loop0: detected capacity change from 0 to 204800000 [10395.955964] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10396.342922] Lustre: sv5644-OST0000: new disk, initializing [10396.350763] Lustre: srv-sv5644-OST0000: No data found on store. Initialize space. [10396.365771] Lustre: 326526:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10396.467328] Lustre: sv5644-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10403.667514] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10416.117718] loop1: detected capacity change from 0 to 204800000 [10436.477106] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10436.722388] Lustre: sv5644-OST0001: new disk, initializing [10436.728995] Lustre: srv-sv5644-OST0001: No data found on store. Initialize space. [10436.749795] Lustre: 327646:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10436.813763] Lustre: sv5644-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10443.892441] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10459.121775] Lustre: server umount sv5644-OST0001 complete [10472.874957] Lustre: server umount sv5644-OST0000 complete [10491.878054] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10491.882332] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10491.893189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10492.387748] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10492.405820] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10492.413305] Lustre: Skipped 1 previous similar message [10494.719370] Lustre: server umount lustre-MDT0000 complete [10499.162579] LustreError: 324438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162264 with bad export cookie 18347753556609232257 [10499.174515] LustreError: 324438:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10499.177419] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10499.920680] Lustre: server umount lustre-MDT0001 complete [10515.775304] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [10519.357843] Key type lgssc unregistered [10519.809634] LNet: 330401:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10519.824888] LNetError: 330401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10519.830167] LNet: Removed LNI 192.168.201.109@tcp [10520.878351] LNet: Removed LNI 192.168.201.109@tcp1 [10522.007918] Key type .llcrypt unregistered [10522.011722] Key type ._llcrypt unregistered [10546.398633] Key type ._llcrypt registered [10546.401245] Key type .llcrypt registered [10546.585527] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [10546.974979] Key type .llcrypt unregistered [10546.981788] Key type ._llcrypt unregistered [10570.113501] Key type ._llcrypt registered [10570.118235] Key type .llcrypt registered [10570.302922] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [10571.463093] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10571.527600] alg: No test for adler32 (adler32-zlib) [10572.963334] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10573.221144] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [10574.815205] Key type lgssc registered [10576.513371] Lustre: Echo OBD driver; http://www.lustre.org/ [10604.488378] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [10606.098442] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [10618.382551] loop0: detected capacity change from 0 to 204800000 [10618.691566] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10618.706839] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10618.722521] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10638.606449] loop0: detected capacity change from 0 to 204800000 [10638.799447] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10638.814281] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10638.836974] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10657.839376] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10657.857494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10659.258517] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10659.281294] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10659.561064] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10665.257080] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10675.502631] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10675.624216] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10675.657445] Lustre: 334226:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10676.014677] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10682.111894] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10692.283810] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10699.167283] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10706.020917] loop0: detected capacity change from 0 to 204800000 [10726.973648] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10727.434563] Lustre: sv21886-OST0000: new disk, initializing [10727.439889] Lustre: srv-sv21886-OST0000: No data found on store. Initialize space. [10727.454184] Lustre: 336308:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10727.552132] Lustre: sv21886-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10735.055896] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10748.814704] loop1: detected capacity change from 0 to 204800000 [10769.972551] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10770.324230] Lustre: sv21886-OST0001: new disk, initializing [10770.332985] Lustre: srv-sv21886-OST0001: No data found on store. Initialize space. [10770.346598] Lustre: 337425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10770.491147] Lustre: sv21886-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10780.303687] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [10797.906735] Lustre: server umount sv21886-OST0001 complete [10809.050721] Lustre: server umount sv21886-OST0000 complete [10824.674902] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10824.689229] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10824.713496] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10825.191874] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10829.199151] Lustre: server umount lustre-MDT0000 complete [10833.831779] LustreError: 334221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162599 with bad export cookie 17771878316616630066 [10833.838819] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10833.848318] LustreError: 334221:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10834.160838] Lustre: server umount lustre-MDT0001 complete [10849.524932] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [10852.848798] Key type lgssc unregistered [10853.199499] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.9@tcp [10853.244781] LNet: 340182:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10853.249886] LNetError: 340182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10853.257287] LNet: Removed LNI 192.168.201.109@tcp [10854.331073] LNet: Removed LNI 192.168.201.109@tcp1 [10855.458490] Key type .llcrypt unregistered [10855.461245] Key type ._llcrypt unregistered [10878.917507] Key type ._llcrypt registered [10878.921489] Key type .llcrypt registered [10879.034841] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [10879.364295] Key type .llcrypt unregistered [10879.368159] Key type ._llcrypt unregistered [10900.058335] Key type ._llcrypt registered [10900.060962] Key type .llcrypt registered [10900.195271] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [10901.720406] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10901.772096] alg: No test for adler32 (adler32-zlib) [10902.879638] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [10903.114073] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [10904.791144] Key type lgssc registered [10906.109881] Lustre: Echo OBD driver; http://www.lustre.org/ [10932.023855] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [10933.574391] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [10947.487694] loop0: detected capacity change from 0 to 204800000 [10947.715324] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10947.728910] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10947.758454] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10968.049346] loop0: detected capacity change from 0 to 204800000 [10968.228477] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10968.239065] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10968.250413] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [10987.979214] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10987.992918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10989.279659] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10989.318958] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10989.599575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10995.030294] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11004.654189] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11004.799816] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11004.859216] Lustre: 344014:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11005.122171] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11010.129804] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11019.313981] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11025.861553] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11031.216789] loop0: detected capacity change from 0 to 204800000 [11050.247985] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11050.813748] Lustre: sv16478-OST0000: new disk, initializing [11050.827764] Lustre: srv-sv16478-OST0000: No data found on store. Initialize space. [11050.837817] Lustre: 346091:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11050.939995] Lustre: sv16478-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11059.586731] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11082.126727] Lustre: server umount sv16478-OST0000 complete [11097.058209] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11097.101442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11102.186170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11102.195025] Lustre: Skipped 2 previous similar messages [11102.935435] Lustre: server umount lustre-MDT0000 complete [11107.996589] LustreError: 344004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788162873 with bad export cookie 8382688502080627549 [11108.013339] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11108.014115] LustreError: 344004:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11108.639450] Lustre: server umount lustre-MDT0001 complete [11125.921747] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [11130.568349] Key type lgssc unregistered [11131.087699] LNet: 348547:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11131.092074] LNetError: 348547:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11131.095791] LNet: Removed LNI 192.168.201.109@tcp [11132.139372] LNet: Removed LNI 192.168.201.109@tcp1 [11133.818781] Key type .llcrypt unregistered [11133.820758] Key type ._llcrypt unregistered [11154.455256] Key type ._llcrypt registered [11154.459326] Key type .llcrypt registered [11154.560840] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [11154.837807] Key type .llcrypt unregistered [11154.839501] Key type ._llcrypt unregistered [11173.215575] Key type ._llcrypt registered [11173.218756] Key type .llcrypt registered [11173.384028] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [11174.957070] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11174.992804] alg: No test for adler32 (adler32-zlib) [11176.029463] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11176.219229] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [11177.850629] Key type lgssc registered [11179.388983] Lustre: Echo OBD driver; http://www.lustre.org/ [11207.401275] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [11208.405941] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [11218.860432] loop0: detected capacity change from 0 to 204800000 [11219.040454] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11219.048897] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11219.056961] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11239.148968] loop0: detected capacity change from 0 to 204800000 [11239.332512] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11239.360230] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11239.378743] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11261.503482] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11261.553429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11263.013924] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11263.038269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11263.302273] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11270.159341] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11279.076325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11279.179299] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11279.216564] Lustre: 352419:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11279.494551] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11284.978676] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11292.785995] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11299.592879] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11304.960102] loop0: detected capacity change from 0 to 204800000 [11323.943351] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11324.121046] Lustre: sv11545-OST0000: new disk, initializing [11324.125607] Lustre: srv-sv11545-OST0000: No data found on store. Initialize space. [11324.138657] Lustre: 354500:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11324.212563] Lustre: sv11545-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11329.711846] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11351.263340] Lustre: server umount sv11545-OST0000 complete [11366.369553] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11366.374201] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11366.410748] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11369.952157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11369.953524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11375.073072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11375.081604] Lustre: Skipped 2 previous similar messages [11380.191264] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11380.196608] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11380.217059] Lustre: Skipped 1 previous similar message [11380.729271] Lustre: server umount lustre-MDT0000 complete [11385.810739] LustreError: 352413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163151 with bad export cookie 12425108393092655957 [11385.823662] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11385.831899] LustreError: 352413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11386.482635] Lustre: server umount lustre-MDT0001 complete [11401.816027] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [11406.091547] Key type lgssc unregistered [11406.668462] LNet: 356953:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11406.684197] LNetError: 356953:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11406.696970] LNet: Removed LNI 192.168.201.109@tcp [11407.717478] LNet: Removed LNI 192.168.201.109@tcp1 [11408.750113] Key type .llcrypt unregistered [11408.752744] Key type ._llcrypt unregistered [11431.398949] Key type ._llcrypt registered [11431.401582] Key type .llcrypt registered [11431.490300] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [11431.797547] Key type .llcrypt unregistered [11431.799259] Key type ._llcrypt unregistered [11453.521600] Key type ._llcrypt registered [11453.524397] Key type .llcrypt registered [11453.736417] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [11455.262424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11455.373877] alg: No test for adler32 (adler32-zlib) [11456.511032] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11456.805528] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [11458.543171] Key type lgssc registered [11460.237866] Lustre: Echo OBD driver; http://www.lustre.org/ [11489.771705] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [11491.107631] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [11505.174928] loop0: detected capacity change from 0 to 204800000 [11505.448396] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11505.457762] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11505.473484] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11523.372173] loop0: detected capacity change from 0 to 204800000 [11523.590282] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11523.617677] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11523.633616] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11542.116684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11542.135199] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11543.437049] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11543.459597] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11543.656941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11548.898585] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11558.877683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11559.005390] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11559.045483] Lustre: 360794:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11559.326979] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11564.688902] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11572.795728] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11579.380280] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11587.094525] loop0: detected capacity change from 0 to 204800000 [11607.918023] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11608.183742] Lustre: sv15793-OST0000: new disk, initializing [11608.185971] Lustre: srv-sv15793-OST0000: No data found on store. Initialize space. [11608.189749] Lustre: 362871:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11608.237704] Lustre: sv15793-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11614.756666] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11629.871756] loop1: detected capacity change from 0 to 204800000 [11648.700876] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11648.922600] Lustre: sv15793-OST0001: new disk, initializing [11648.929054] Lustre: srv-sv15793-OST0001: No data found on store. Initialize space. [11648.941792] Lustre: 364042:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11649.070496] Lustre: sv15793-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11656.768748] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11671.961269] Lustre: server umount sv15793-OST0001 complete [11683.935697] Lustre: server umount sv15793-OST0000 complete [11702.752275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11702.755241] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11702.773227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11703.264386] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11703.282962] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11703.291573] Lustre: Skipped 1 previous similar message [11705.293877] Lustre: server umount lustre-MDT0000 complete [11709.119562] LustreError: 360785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163474 with bad export cookie 510407650526360474 [11709.123837] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11709.127891] LustreError: 360785:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11709.459863] Lustre: server umount lustre-MDT0001 complete [11721.923642] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [11724.632454] Key type lgssc unregistered [11724.931892] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.9@tcp [11724.963830] LNet: 366796:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11724.967617] LNetError: 366796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11725.984718] LNet: Removed LNI 192.168.201.109@tcp [11725.987572] LNetError: 366796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11727.019080] LNet: Removed LNI 192.168.201.109@tcp1 [11728.195299] Key type .llcrypt unregistered [11728.199961] Key type ._llcrypt unregistered [11749.202153] Key type ._llcrypt registered [11749.204349] Key type .llcrypt registered [11749.299842] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [11749.510446] Key type .llcrypt unregistered [11749.511680] Key type ._llcrypt unregistered [11769.688581] Key type ._llcrypt registered [11769.692645] Key type .llcrypt registered [11769.869627] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [11771.040986] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11771.239386] alg: No test for adler32 (adler32-zlib) [11772.414136] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11772.577304] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [11774.223258] Key type lgssc registered [11775.847829] Lustre: Echo OBD driver; http://www.lustre.org/ [11798.669183] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [11800.099641] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [11835.593230] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [11839.177096] Key type lgssc unregistered [11839.456234] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.9@tcp [11839.506335] LNet: 370759:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11839.522336] LNetError: 370759:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11839.529675] LNet: Removed LNI 192.168.201.109@tcp [11839.562294] LNet: Removed LNI 192.168.201.109@tcp1 [11840.734688] Key type .llcrypt unregistered [11840.737329] Key type ._llcrypt unregistered [11863.079282] Key type ._llcrypt registered [11863.081567] Key type .llcrypt registered [11863.289385] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [11863.735764] Key type .llcrypt unregistered [11863.739446] Key type ._llcrypt unregistered [11883.262915] Key type ._llcrypt registered [11883.266838] Key type .llcrypt registered [11883.367847] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [11885.222678] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11885.313048] alg: No test for adler32 (adler32-zlib) [11886.470888] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [11886.905214] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [11888.599233] Key type lgssc registered [11889.811980] Lustre: Echo OBD driver; http://www.lustre.org/ [11910.380750] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [11911.606225] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [11913.038543] LNet: Added LNI 192.168.201.109@tcp2 [8/256/0/180] [11924.871280] loop0: detected capacity change from 0 to 204800000 [11925.177731] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11925.191607] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11925.211602] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11942.667870] loop0: detected capacity change from 0 to 204800000 [11942.951666] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11942.960814] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11942.985643] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [11962.921602] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11962.941869] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11964.103345] LNetError: 372087:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.201.109@tcp NID 192.168.201.109@tcp is already locked with peer 192.168.201.109@tcp [11964.242477] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [11964.268664] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11964.420159] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11969.556849] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11979.682501] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11979.784166] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [11979.813667] Lustre: 374675:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11979.960723] LNetError: 372087:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.201.109@tcp NID 192.168.201.109@tcp is already locked with peer 192.168.201.109@tcp [11979.972986] LNetError: 372087:0:(peer.c:2151:lnet_peer_add_nid()) Skipped 1 previous similar message [11980.054880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11983.898061] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [11989.047984] LNetError: 372087:0:(peer.c:2151:lnet_peer_add_nid()) Peer 192.168.201.109@tcp NID 192.168.201.109@tcp is already locked with peer 192.168.201.109@tcp [11989.058092] LNetError: 372087:0:(peer.c:2151:lnet_peer_add_nid()) Skipped 1 previous similar message [11991.512648] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11997.727407] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12003.404816] loop0: detected capacity change from 0 to 204800000 [12023.084573] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12023.657579] Lustre: sv10182-OST0000: new disk, initializing [12023.667515] Lustre: srv-sv10182-OST0000: No data found on store. Initialize space. [12023.676129] Lustre: 376752:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12023.817310] Lustre: sv10182-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12032.951223] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12046.060402] loop1: detected capacity change from 0 to 204800000 [12064.690793] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12065.009754] Lustre: sv10182-OST0001: new disk, initializing [12065.027903] Lustre: srv-sv10182-OST0001: No data found on store. Initialize space. [12065.043377] Lustre: 377922:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12065.187990] Lustre: sv10182-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12071.797729] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12084.858822] Lustre: server umount sv10182-OST0001 complete [12093.655992] Lustre: server umount sv10182-OST0000 complete [12106.720758] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12106.723630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12106.743839] Lustre: Skipped 1 previous similar message [12111.840403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12111.854287] Lustre: Skipped 2 previous similar messages [12112.259069] Lustre: server umount lustre-MDT0000 complete [12116.316647] LustreError: 374668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788163881 with bad export cookie 1790572517667982533 [12116.323529] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12116.329188] LustreError: 374668:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12116.728845] Lustre: server umount lustre-MDT0001 complete [12130.339093] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [12133.370626] Key type lgssc unregistered [12133.676067] LNet: 380676:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12133.687435] LNetError: 380676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12133.698314] LNet: Removed LNI 192.168.201.109@tcp [12134.751252] LNet: Removed LNI 192.168.201.109@tcp1 [12134.753492] LNetError: 380676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12134.763542] LNetError: 380676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12134.783601] LNet: Removed LNI 192.168.201.109@tcp2 [12135.651086] Key type .llcrypt unregistered [12135.655172] Key type ._llcrypt unregistered [12155.483830] Key type ._llcrypt registered [12155.485653] Key type .llcrypt registered [12155.556546] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [12155.818571] Key type .llcrypt unregistered [12155.828651] Key type ._llcrypt unregistered [12173.527505] Key type ._llcrypt registered [12173.528776] Key type .llcrypt registered [12173.647188] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [12174.772381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12174.893156] alg: No test for adler32 (adler32-zlib) [12176.149419] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12176.322429] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [12178.032398] Key type lgssc registered [12179.659739] Lustre: Echo OBD driver; http://www.lustre.org/ [12200.423155] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [12201.759565] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [12212.451118] loop0: detected capacity change from 0 to 204800000 [12212.595214] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12212.605370] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12212.629633] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12231.046957] loop0: detected capacity change from 0 to 204800000 [12231.298440] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12231.307451] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12231.343320] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12250.366526] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12250.397430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12251.776914] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12251.802482] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12252.030464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12256.881414] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12266.286843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12266.359314] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12266.392777] Lustre: 384547:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12266.648333] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12272.455069] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12280.925975] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12288.469995] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12295.133538] loop0: detected capacity change from 0 to 204800000 [12315.366408] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12315.661232] Lustre: sv12595-OST0000: new disk, initializing [12315.666244] Lustre: srv-sv12595-OST0000: No data found on store. Initialize space. [12315.674402] Lustre: 386626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12315.728591] Lustre: sv12595-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12323.598854] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12333.590071] loop1: detected capacity change from 0 to 204800000 [12352.548077] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12352.809553] Lustre: sv12595-OST0001: new disk, initializing [12352.813622] Lustre: srv-sv12595-OST0001: No data found on store. Initialize space. [12352.820130] Lustre: 387745:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12352.870174] Lustre: sv12595-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12358.531154] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12371.699447] Lustre: server umount sv12595-OST0001 complete [12381.226782] Lustre: server umount sv12595-OST0000 complete [12394.976618] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12395.001293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12395.032695] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12396.004402] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12396.007830] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12399.873921] Lustre: server umount lustre-MDT0000 complete [12403.705422] LustreError: 384540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164169 with bad export cookie 16678964508619872335 [12403.710722] LustreError: 384540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12403.717158] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12404.019140] Lustre: server umount lustre-MDT0001 complete [12416.529738] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [12419.138827] Key type lgssc unregistered [12419.418528] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.9@tcp [12419.466776] LNet: 390499:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12419.472626] LNetError: 390499:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12419.481134] LNet: Removed LNI 192.168.201.109@tcp [12420.526891] LNet: Removed LNI 192.168.201.109@tcp1 [12421.872564] Key type .llcrypt unregistered [12421.875992] Key type ._llcrypt unregistered [12442.562655] Key type ._llcrypt registered [12442.570029] Key type .llcrypt registered [12442.735394] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [12443.002415] Key type .llcrypt unregistered [12443.004803] Key type ._llcrypt unregistered [12450.538963] Key type ._llcrypt registered [12450.540586] Key type .llcrypt registered [12450.642642] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [12450.925800] Key type .llcrypt unregistered [12450.933254] Key type ._llcrypt unregistered [12465.039848] Key type ._llcrypt registered [12465.043867] Key type .llcrypt registered [12465.137786] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [12465.359527] Key type .llcrypt unregistered [12465.363673] Key type ._llcrypt unregistered [12483.172792] Key type ._llcrypt registered [12483.181711] Key type .llcrypt registered [12483.338062] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [12483.747530] Key type .llcrypt unregistered [12483.750717] Key type ._llcrypt unregistered [12501.687882] Key type ._llcrypt registered [12501.692038] Key type .llcrypt registered [12501.857428] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [12503.210149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12503.240310] alg: No test for adler32 (adler32-zlib) [12504.356809] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12504.565202] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [12506.239922] Key type lgssc registered [12507.426849] Lustre: Echo OBD driver; http://www.lustre.org/ [12528.594438] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [12529.944860] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [12540.282104] loop0: detected capacity change from 0 to 204800000 [12540.403299] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12540.424092] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12540.442034] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12559.100193] loop0: detected capacity change from 0 to 204800000 [12559.471867] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12559.479874] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12559.502044] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12581.617284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12581.638805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12583.154488] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12583.198909] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12583.506207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12588.863871] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12598.438266] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12598.525214] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12598.562936] Lustre: 395956:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12598.761470] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12602.451261] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12609.647667] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12615.191561] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12620.233926] loop0: detected capacity change from 0 to 204800000 [12638.200149] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12638.484990] Lustre: sv8604-OST0000: new disk, initializing [12638.490621] Lustre: srv-sv8604-OST0000: No data found on store. Initialize space. [12638.502274] Lustre: 398032:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12638.576159] Lustre: sv8604-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12645.619389] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12658.116983] loop1: detected capacity change from 0 to 204800000 [12676.982522] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12677.364063] Lustre: sv8604-OST0001: new disk, initializing [12677.383305] Lustre: srv-sv8604-OST0001: No data found on store. Initialize space. [12677.397845] Lustre: 399149:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12677.528666] Lustre: sv8604-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12684.436868] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12702.139549] Lustre: 400141:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [12702.155137] Lustre: 400141:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [12702.161576] Lustre: 400141:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [12702.174179] Lustre: 400141:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [12702.180617] Lustre: 400141:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [12702.186882] Lustre: 400141:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [12704.857508] Lustre: server umount sv8604-OST0001 complete [12718.128519] Lustre: server umount sv8604-OST0000 complete [12735.456957] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12735.475633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12735.485164] Lustre: Skipped 1 previous similar message [12739.297045] Lustre: server umount lustre-MDT0000 complete [12743.521712] LustreError: 395948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164509 with bad export cookie 9461192207536140869 [12743.526598] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12743.532225] LustreError: 395948:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12743.885483] Lustre: server umount lustre-MDT0001 complete [12758.740182] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [12762.220837] Key type lgssc unregistered [12762.485216] LNet: 402148:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12762.492536] LNetError: 402148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12762.501267] LNet: Removed LNI 192.168.201.109@tcp [12763.558605] LNet: Removed LNI 192.168.201.109@tcp1 [12764.553121] Key type .llcrypt unregistered [12764.558075] Key type ._llcrypt unregistered [12784.377807] Key type ._llcrypt registered [12784.382503] Key type .llcrypt registered [12784.539084] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [12784.841485] Key type .llcrypt unregistered [12784.849769] Key type ._llcrypt unregistered [12805.409189] Key type ._llcrypt registered [12805.414070] Key type .llcrypt registered [12805.537290] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [12806.632376] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12806.809927] alg: No test for adler32 (adler32-zlib) [12807.878785] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [12808.223489] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [12809.879484] Key type lgssc registered [12811.071981] Lustre: Echo OBD driver; http://www.lustre.org/ [12833.785311] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [12834.923312] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [12847.126799] loop0: detected capacity change from 0 to 204800000 [12847.404018] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12847.427098] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12847.440706] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12865.547803] loop0: detected capacity change from 0 to 204800000 [12865.743919] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12865.763225] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12865.773994] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12885.480209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12885.509882] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12886.834304] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12886.876039] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12887.118009] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12892.364153] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12902.587940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12902.732766] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12902.785797] Lustre: 405974:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12903.098696] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12907.990729] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12916.594351] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12924.177895] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12930.603437] loop0: detected capacity change from 0 to 204800000 [12949.668339] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12950.099630] Lustre: sv31138-OST0000: new disk, initializing [12950.104053] Lustre: srv-sv31138-OST0000: No data found on store. Initialize space. [12950.109843] Lustre: 408052:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12950.163878] Lustre: sv31138-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12958.479697] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [12968.997181] loop1: detected capacity change from 0 to 204800000 [12988.213668] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12988.407442] Lustre: sv31138-OST0001: new disk, initializing [12988.410944] Lustre: srv-sv31138-OST0001: No data found on store. Initialize space. [12988.416261] Lustre: 409170:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12988.456807] Lustre: sv31138-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12994.949590] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [13012.197731] Lustre: 410162:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13012.211470] Lustre: 410162:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13012.222945] Lustre: 410162:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13012.228890] Lustre: 410162:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13012.237841] Lustre: 410162:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13012.244702] Lustre: 410162:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13014.680064] Lustre: server umount sv31138-OST0001 complete [13027.631783] Lustre: server umount sv31138-OST0000 complete [13046.755820] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13046.761284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13046.810294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13046.878561] Lustre: Skipped 1 previous similar message [13050.460835] Lustre: server umount lustre-MDT0000 complete [13055.331296] LustreError: 406715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788164820 with bad export cookie 14452453424439987506 [13055.343320] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13055.346051] LustreError: 406715:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13055.862569] Lustre: server umount lustre-MDT0001 complete [13071.623930] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [13075.087911] Key type lgssc unregistered [13075.323062] LNet: 412168:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13075.338800] LNetError: 412168:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13075.349516] LNet: Removed LNI 192.168.201.109@tcp [13076.393527] LNet: Removed LNI 192.168.201.109@tcp1 [13077.491954] Key type .llcrypt unregistered [13077.494257] Key type ._llcrypt unregistered [13098.961663] Key type ._llcrypt registered [13098.964547] Key type .llcrypt registered [13099.129380] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [13099.509947] Key type .llcrypt unregistered [13099.513979] Key type ._llcrypt unregistered [13120.255388] Key type ._llcrypt registered [13120.260507] Key type .llcrypt registered [13120.369840] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [13121.678141] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13121.692953] alg: No test for adler32 (adler32-zlib) [13122.800517] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13123.162755] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [13124.898271] Key type lgssc registered [13126.268983] Lustre: Echo OBD driver; http://www.lustre.org/ [13148.275509] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [13149.537122] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [13159.431560] loop0: detected capacity change from 0 to 204800000 [13159.534432] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13159.550329] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13159.575583] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13177.119464] loop0: detected capacity change from 0 to 204800000 [13177.349357] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13177.364977] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13177.378148] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13196.064480] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13196.101969] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13197.371227] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13197.415681] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13197.568907] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13202.546680] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [13211.922803] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13212.035763] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13212.075891] Lustre: 415998:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13212.320043] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13216.663181] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [13224.480786] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13230.233524] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13235.069688] loop0: detected capacity change from 0 to 204800000 [13254.261286] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13254.620711] Lustre: sv27713-OST0000: new disk, initializing [13254.626182] Lustre: srv-sv27713-OST0000: No data found on store. Initialize space. [13254.637611] Lustre: 418074:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13254.713992] Lustre: sv27713-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13260.989458] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [13270.699772] loop1: detected capacity change from 0 to 204800000 [13290.091388] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13290.377160] Lustre: sv27713-OST0001: new disk, initializing [13290.379859] Lustre: srv-sv27713-OST0001: No data found on store. Initialize space. [13290.385389] Lustre: 419191:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13290.430270] Lustre: sv27713-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13296.401143] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [13314.495270] Lustre: 420278:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13314.499617] Lustre: 420278:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13314.503859] Lustre: 420278:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13314.509177] Lustre: 420278:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13314.513794] Lustre: 420278:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13314.517898] Lustre: 420278:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13315.257267] Lustre: 420326:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13315.263632] Lustre: 420326:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 255 previous similar messages [13315.268151] Lustre: 420326:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13315.272979] Lustre: 420326:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 255 previous similar messages [13315.279469] Lustre: 420326:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13315.284219] Lustre: 420326:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 255 previous similar messages [13315.288820] Lustre: 420326:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13315.292555] Lustre: 420326:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 255 previous similar messages [13315.296473] Lustre: 420326:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13315.300395] Lustre: 420326:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 255 previous similar messages [13315.305256] Lustre: 420326:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13315.309921] Lustre: 420326:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 255 previous similar messages [13317.246978] Lustre: server umount sv27713-OST0001 complete [13326.037301] Lustre: server umount sv27713-OST0000 complete [13340.128670] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13340.148518] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13340.168249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13342.691524] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13342.695890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13342.741397] Lustre: Skipped 1 previous similar message [13344.743445] Lustre: server umount lustre-MDT0000 complete [13349.325613] LustreError: 416749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165114 with bad export cookie 11862947166487185324 [13349.331699] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13349.339892] LustreError: 416749:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13349.669880] Lustre: server umount lustre-MDT0001 complete [13366.270864] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [13368.877923] Key type lgssc unregistered [13369.232789] LNet: 422346:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13369.238084] LNetError: 422346:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13369.242481] LNet: Removed LNI 192.168.201.109@tcp [13370.285312] LNet: Removed LNI 192.168.201.109@tcp1 [13371.312276] Key type .llcrypt unregistered [13371.314046] Key type ._llcrypt unregistered [13391.492898] Key type ._llcrypt registered [13391.495807] Key type .llcrypt registered [13391.621087] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [13392.066366] Key type .llcrypt unregistered [13392.067653] Key type ._llcrypt unregistered [13415.199043] Key type ._llcrypt registered [13415.201082] Key type .llcrypt registered [13415.299572] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [13415.858706] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13415.875449] alg: No test for adler32 (adler32-zlib) [13416.902624] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13417.128364] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [13418.775124] Key type lgssc registered [13420.165806] Lustre: Echo OBD driver; http://www.lustre.org/ [13444.438731] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [13445.919996] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [13457.380372] loop0: detected capacity change from 0 to 204800000 [13457.557950] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13457.582696] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13457.598199] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13477.499341] loop0: detected capacity change from 0 to 204800000 [13477.824574] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13477.836261] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13477.849806] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13498.727074] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13498.741601] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13500.063326] LNet: 1 peer NIs in recovery (showing 1): 192.168.201.109@tcp [13500.096972] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13500.162771] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13500.379584] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13506.171303] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [13515.976932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13516.043309] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13516.083255] Lustre: 426170:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13516.363823] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13521.197495] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [13530.341816] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13537.137113] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13542.387950] loop0: detected capacity change from 0 to 204800000 [13562.556932] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13562.883595] Lustre: sv3383-OST0000: new disk, initializing [13562.887143] Lustre: srv-sv3383-OST0000: No data found on store. Initialize space. [13562.905953] Lustre: 428249:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13562.979894] Lustre: sv3383-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13568.558535] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [13584.665142] Lustre: 429190:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13584.686441] Lustre: 429190:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13584.699957] Lustre: 429190:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13584.710607] Lustre: 429190:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13584.718196] Lustre: 429190:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13584.729641] Lustre: 429190:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13590.735839] Lustre: server umount sv3383-OST0000 complete [13604.323124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13604.344810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13604.359138] Lustre: Skipped 1 previous similar message [13609.442165] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13610.808525] Lustre: server umount lustre-MDT0000 complete [13615.909868] LustreError: 426163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165381 with bad export cookie 17774429812064266736 [13615.914297] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13615.924785] LustreError: 426163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13616.338883] Lustre: server umount lustre-MDT0001 complete [13630.753894] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [13633.692695] Key type lgssc unregistered [13634.099851] LNet: 430896:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13634.111721] LNetError: 430896:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13634.123121] LNet: Removed LNI 192.168.201.109@tcp [13635.205577] LNet: Removed LNI 192.168.201.109@tcp1 [13636.095101] Key type .llcrypt unregistered [13636.097153] Key type ._llcrypt unregistered [13657.692577] Key type ._llcrypt registered [13657.696237] Key type .llcrypt registered [13657.857858] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [13658.226318] Key type .llcrypt unregistered [13658.230177] Key type ._llcrypt unregistered [13676.902618] Key type ._llcrypt registered [13676.906617] Key type .llcrypt registered [13677.072754] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [13678.527835] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13678.610294] alg: No test for adler32 (adler32-zlib) [13679.616453] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13679.950242] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [13681.639691] Key type lgssc registered [13682.520774] Lustre: Echo OBD driver; http://www.lustre.org/ [13701.505096] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [13702.880518] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [13704.610549] LNet: Added LNI 192.168.201.109@tcp2 [8/256/0/180] [13715.951400] loop0: detected capacity change from 0 to 204800000 [13716.154919] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13716.168629] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13716.182594] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13733.894554] loop0: detected capacity change from 0 to 204800000 [13734.117845] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13734.128521] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13734.137385] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13752.509901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13752.527883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13753.863096] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13753.895545] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13754.086286] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13758.806899] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [13766.855309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13767.025645] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13767.087699] Lustre: 434819:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13767.432322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13771.884337] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [13779.512271] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13786.180459] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13793.142427] loop0: detected capacity change from 0 to 204800000 [13813.613172] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13813.932454] Lustre: sv3298-OST0000: new disk, initializing [13813.935581] Lustre: srv-sv3298-OST0000: No data found on store. Initialize space. [13813.946377] Lustre: 436897:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13814.004497] Lustre: sv3298-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13822.059672] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [13835.811738] loop1: detected capacity change from 0 to 204800000 [13855.672093] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13855.874188] Lustre: sv3298-OST0001: new disk, initializing [13855.880625] Lustre: srv-sv3298-OST0001: No data found on store. Initialize space. [13855.885935] Lustre: 438016:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13856.008737] Lustre: sv3298-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13862.799598] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [13872.350575] Lustre: DEBUG MARKER: SKIP: conf-sanity test_57p need a remote client with no Lustre server role for routing [13873.592886] Lustre: 438764:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [13873.600246] Lustre: 438764:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [13873.605286] Lustre: 438764:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [13873.611384] Lustre: 438764:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [13873.620151] Lustre: 438764:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [13873.626935] Lustre: 438764:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [13875.746369] Lustre: server umount sv3298-OST0001 complete [13884.939291] Lustre: server umount sv3298-OST0000 complete [13900.768619] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13900.773513] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13900.790567] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13902.308447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13902.309287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13902.327400] Lustre: Skipped 1 previous similar message [13905.880605] Lustre: server umount lustre-MDT0000 complete [13909.779813] LustreError: 435559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788165675 with bad export cookie 5831069932911489568 [13909.784528] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13909.793362] LustreError: 435559:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13910.154847] Lustre: server umount lustre-MDT0001 complete [13923.696991] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [13926.633269] Key type lgssc unregistered [13927.038669] LNet: 440769:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13927.048892] LNetError: 440769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13927.069837] LNet: Removed LNI 192.168.201.109@tcp [13928.095899] LNet: Removed LNI 192.168.201.109@tcp1 [13928.116242] LNetError: 440769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13928.129972] LNetError: 440769:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13928.161898] LNet: Removed LNI 192.168.201.109@tcp2 [13929.223362] Key type .llcrypt unregistered [13929.229400] Key type ._llcrypt unregistered [13953.033425] Key type ._llcrypt registered [13953.037723] Key type .llcrypt registered [13953.174584] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [13953.488654] Key type .llcrypt unregistered [13953.491443] Key type ._llcrypt unregistered [13973.320728] Key type ._llcrypt registered [13973.325420] Key type .llcrypt registered [13973.564106] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [13975.004198] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13975.032745] alg: No test for adler32 (adler32-zlib) [13976.209091] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [13976.494939] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [13978.159445] Key type lgssc registered [13979.044719] Lustre: Echo OBD driver; http://www.lustre.org/ [14002.685178] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [14004.166943] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [14015.002912] loop0: detected capacity change from 0 to 204800000 [14015.269325] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14015.288910] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14015.298373] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14032.667587] loop0: detected capacity change from 0 to 204800000 [14032.889353] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14032.900338] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14032.918209] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14051.923891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14051.944595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14053.326422] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [14053.352260] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14053.519928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14058.454250] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14066.614921] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14066.703467] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [14066.724674] Lustre: 444642:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14066.948492] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14071.507791] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14079.214574] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14086.332302] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14091.983312] loop0: detected capacity change from 0 to 204800000 [14110.635462] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14110.984433] Lustre: sv9254-OST0000: new disk, initializing [14110.987847] Lustre: srv-sv9254-OST0000: No data found on store. Initialize space. [14110.993372] Lustre: 446720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14111.052271] Lustre: sv9254-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14117.749933] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14128.505285] loop1: detected capacity change from 0 to 204800000 [14147.765506] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14147.987778] Lustre: sv9254-OST0001: new disk, initializing [14148.000034] Lustre: srv-sv9254-OST0001: No data found on store. Initialize space. [14148.009471] Lustre: 447840:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14148.088235] Lustre: sv9254-OST0001: Imperative Recovery not enabled, recovery window 60-180 [14155.986673] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14169.819914] LustreError: 448736:0:(nodemap_handler.c:262:nodemap_disc_visibility_update_end_nm()) cfs_fail_timeout id 2404 sleeping for 15000ms [14184.920055] LustreError: 448736:0:(nodemap_handler.c:262:nodemap_disc_visibility_update_end_nm()) cfs_fail_timeout id 2404 awake [14196.907419] Lustre: 449030:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [14196.913708] Lustre: 449030:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [14196.931322] Lustre: 449030:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [14196.951876] Lustre: 449030:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [14196.958913] Lustre: 449030:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [14196.973101] Lustre: 449030:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [14199.689411] Lustre: server umount sv9254-OST0001 complete [14210.158321] Lustre: server umount sv9254-OST0000 complete [14225.893330] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14225.898152] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14225.906758] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14227.939939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14227.946437] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14230.852626] Lustre: server umount lustre-MDT0000 complete [14235.757738] LustreError: 444634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166001 with bad export cookie 3737186804930768762 [14235.776441] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14235.782054] LustreError: 444634:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14236.303402] Lustre: server umount lustre-MDT0001 complete [14254.935874] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [14259.290941] Key type lgssc unregistered [14259.751446] LNet: 451058:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14259.761508] LNetError: 451058:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14259.781264] LNet: Removed LNI 192.168.201.109@tcp [14260.880564] LNet: Removed LNI 192.168.201.109@tcp1 [14262.463614] Key type .llcrypt unregistered [14262.465863] Key type ._llcrypt unregistered [14288.262165] Key type ._llcrypt registered [14288.267038] Key type .llcrypt registered [14288.444938] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [14288.867423] Key type .llcrypt unregistered [14288.873561] Key type ._llcrypt unregistered [14308.508462] Key type ._llcrypt registered [14308.510651] Key type .llcrypt registered [14308.609857] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [14309.753942] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14309.936415] alg: No test for adler32 (adler32-zlib) [14311.157458] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14311.510199] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [14313.239249] Key type lgssc registered [14314.762295] Lustre: Echo OBD driver; http://www.lustre.org/ [14346.748173] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [14348.071232] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [14360.907871] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14360.944928] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14362.250111] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [14362.281485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14362.510845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14368.359037] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14379.694170] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14379.835922] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [14379.860352] Lustre: 454331:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14380.152190] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14386.717217] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14400.024902] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14409.379140] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14414.546348] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [14415.034901] Lustre: DEBUG MARKER: rpc test_57r: @@@@@@ FAIL: Can't determine interface name for NID 192.168.201.109@tcp1 [14416.864840] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing check_logdir /tmp/test_logs/1788166179 [14425.568922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14425.584657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14425.600384] Lustre: Skipped 1 previous similar message [14430.690104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14430.703771] Lustre: Skipped 1 previous similar message [14431.769807] Lustre: server umount lustre-MDT0000 complete [14436.210830] LustreError: 454325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166201 with bad export cookie 13603638763454247567 [14436.213311] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14436.217301] LustreError: 454325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14436.957891] Lustre: server umount lustre-MDT0001 complete [14455.427504] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [14459.459503] Key type lgssc unregistered [14459.704893] LNet: 457681:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14459.711372] LNetError: 457681:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14459.718737] LNet: Removed LNI 192.168.201.109@tcp [14460.775517] LNet: Removed LNI 192.168.201.109@tcp1 [14461.436142] Key type .llcrypt unregistered [14461.442502] Key type ._llcrypt unregistered [14483.376425] Key type ._llcrypt registered [14483.384478] Key type .llcrypt registered [14483.570708] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [14484.725963] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14484.762748] alg: No test for adler32 (adler32-zlib) [14486.461504] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14486.800539] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [14488.599520] Key type lgssc registered [14490.685808] Lustre: Echo OBD driver; http://www.lustre.org/ [14500.704483] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [14502.421979] LNet: Added LNI 192.168.201.109@tcp1 [8/256/0/180] [14511.517777] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14511.536690] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14512.972779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14518.065932] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14523.365837] LustreError: 459163:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14523.386471] LustreError: 459163:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14528.491384] LustreError: 459164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14529.758720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14530.161508] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14534.886458] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14543.824941] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14552.039716] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14561.631213] LNet: Removed LNI 192.168.201.109@tcp1 [14563.772170] Lustre: 460874:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 9: before 516 < left 1024, rollback = 8 [14563.779358] Lustre: 460874:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 0/0/0, destroy: 0/0/0 [14563.782360] Lustre: 460874:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 0/0/0, xattr_set: 1/1/0 [14563.786182] Lustre: 460874:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/0 [14563.789406] Lustre: 460874:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 0/0/0, delete: 256/1024/0 [14563.797350] Lustre: 460874:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [14570.976824] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14570.985819] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14571.004092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14571.495342] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14571.514422] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14571.521382] Lustre: Skipped 1 previous similar message [14576.610956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14576.666766] LustreError: 459158:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14578.723767] Lustre: server umount lustre-MDT0000 complete [14581.731410] LustreError: 459890:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14581.759088] LustreError: 459890:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14583.585774] LustreError: 459143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166349 with bad export cookie 13338122154478981515 [14583.589784] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14583.596559] LustreError: 459143:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14584.083617] Lustre: server umount lustre-MDT0001 complete [14598.352569] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [14602.197923] Key type lgssc unregistered [14602.510699] LNet: 461880:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14602.523129] LNetError: 461880:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14602.550335] LNet: Removed LNI 192.168.201.109@tcp [14603.696232] Key type .llcrypt unregistered [14603.707791] Key type ._llcrypt unregistered [14631.920503] Key type ._llcrypt registered [14631.926235] Key type .llcrypt registered [14632.092784] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [14633.654161] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14633.738411] alg: No test for adler32 (adler32-zlib) [14634.749483] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14634.953174] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [14636.607275] Key type lgssc registered [14637.909535] Lustre: Echo OBD driver; http://www.lustre.org/ [14647.403098] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14647.417026] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14649.025789] LNetError: 462512:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.109@tcp1 has no available nets [14649.099316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14654.940411] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14659.562689] LustreError: 463203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14659.587785] LustreError: 463203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14664.676456] LustreError: 463204:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14664.969967] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14665.358983] LNetError: 462512:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.109@tcp1 has no available nets [14665.375091] LNetError: 462512:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14665.523544] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14670.087706] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14678.707847] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14685.418066] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14694.970962] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14695.129632] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [14695.322895] LNetError: 462512:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.109@tcp1 has no available nets [14695.335126] LNetError: 462512:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14695.386449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14702.183156] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14703.661404] LNetError: 462512:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.109@tcp1 has no available nets [14703.681776] LNetError: 462512:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14710.380665] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14719.460234] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14719.478728] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14719.479115] Lustre: Skipped 1 previous similar message [14719.496303] Lustre: Skipped 2 previous similar messages [14722.479149] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping) [14724.580928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14724.591043] Lustre: Skipped 2 previous similar messages [14725.484879] LustreError: 464540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14725.569606] Lustre: server umount lustre-MDT0000 complete [14729.700811] LustreError: 463203:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14729.716506] LustreError: 463203:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14729.986630] LustreError: 463183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166495 with bad export cookie 17917695522322268227 [14729.997627] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14730.001388] LustreError: 463183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14730.650790] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [14734.818840] LustreError: 465708:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14734.820621] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14734.865288] LustreError: 465708:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14734.913438] Lustre: Skipped 1 previous similar message [14734.921653] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14736.446103] Lustre: server umount lustre-MDT0001 complete [14744.516259] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [14754.667628] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14756.000166] LustreError: 466796:0:(import.c:339:ptlrpc_invalidate_import()) MGS: timeout waiting for callback (1 != 0) [14756.010865] LustreError: 466796:0:(import.c:363:ptlrpc_invalidate_import()) @@@ still on sending list req@00000000ab5e2f18 x1875028371087744/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 1788166521 ref 1 fl Rpc:NQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14756.044458] LustreError: 466796:0:(import.c:373:ptlrpc_invalidate_import()) MGS: Unregistering RPCs found (0). Network is sluggish? Waiting for them to error out. [14756.255854] LustreError: 462513:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000027747d50 x1875028371088000/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [14756.800250] LNetError: 462512:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.109@tcp1 has no available nets [14756.809806] LNetError: 462512:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [14756.841018] LustreError: 465032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14756.865152] LustreError: 465032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [14756.936444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14757.029924] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [14761.662698] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14765.034415] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14771.232860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14771.535290] LNetError: 462512:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.109@tcp1 has no available nets [14771.552759] LNetError: 462512:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [14771.668607] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14774.797666] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [14777.154784] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14787.735370] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14789.915921] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [14797.850884] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14800.077495] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [14807.520082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14807.528173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14807.552177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14807.569041] Lustre: Skipped 1 previous similar message [14808.033636] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14813.195962] Lustre: server umount lustre-OST0000 complete [14817.766234] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14817.772259] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14817.781758] Lustre: Skipped 1 previous similar message [14823.913988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14823.929565] Lustre: Skipped 6 previous similar messages [14832.095142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14832.508625] Lustre: server umount lustre-MDT0000 complete [14834.147123] LustreError: 467014:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14834.167543] LustreError: 467014:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [14837.304655] LustreError: 463183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166602 with bad export cookie 17917695522322275829 [14837.310579] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14837.315401] LustreError: 463183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14837.699732] Lustre: server umount lustre-MDT0001 complete [14853.004201] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [14856.513338] Key type lgssc unregistered [14856.849658] LNet: 469485:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14856.860051] LNetError: 469485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14856.884699] LNet: Removed LNI 192.168.201.109@tcp [14858.108355] Key type .llcrypt unregistered [14858.112776] Key type ._llcrypt unregistered [14886.666425] Key type ._llcrypt registered [14886.678872] Key type .llcrypt registered [14886.877427] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [14888.028894] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14888.108486] alg: No test for adler32 (adler32-zlib) [14889.631418] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [14889.894571] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [14891.559753] Key type lgssc registered [14892.860495] Lustre: Echo OBD driver; http://www.lustre.org/ [14903.661971] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14903.670300] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14905.009839] LNetError: 470209:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.109@tcp1 has no available nets [14905.051670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14910.423393] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14914.532202] LustreError: 470901:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14914.542328] LustreError: 470901:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14919.659621] LustreError: 470900:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14920.251867] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14920.538106] LNetError: 470209:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.109@tcp1 has no available nets [14920.548205] LNetError: 470209:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 4 previous similar messages [14920.671887] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14925.480826] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14934.117688] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14942.014687] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14946.276139] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14946.281566] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14946.297875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14947.364892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14947.375035] Lustre: Skipped 2 previous similar messages [14952.430483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14952.444266] Lustre: Skipped 1 previous similar message [14952.811574] Lustre: server umount lustre-MDT0000 complete [14956.267260] LustreError: 470881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166721 with bad export cookie 12081216710023955880 [14956.281478] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14956.291221] LustreError: 470881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14956.708828] Lustre: server umount lustre-MDT0001 complete [14973.165575] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14973.361591] LNetError: 470209:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.109@tcp1 has no available nets [14973.375462] LNetError: 470209:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 3 previous similar messages [14973.566930] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [14973.599953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14973.823614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14979.972551] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [14988.957542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14989.094883] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [14989.130980] Lustre: 473490:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14989.346734] LNetError: 470209:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.109@tcp1 has no available nets [14989.352051] LNetError: 470209:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 1 previous similar message [14989.454042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14994.626528] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15002.727631] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15009.958124] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15021.630369] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15022.021533] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [15022.041082] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [15022.059828] LustreError: 475376:0:(tgt_mount.c:2688:server_fill_super()) Unable to start targets: -2 [15022.075744] LustreError: 475376:0:(tgt_mount.c:2206:server_put_super()) no obd lustre-OST0000 [15022.084783] LustreError: 475376:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [15022.292780] Lustre: server umount lustre-OST0000 complete [15022.300074] LustreError: 475376:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [15037.496827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15037.766057] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [15038.112748] LNetError: 470209:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.109@tcp1 has no available nets [15038.127737] LNetError: 470209:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 2 previous similar messages [15038.222750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15043.296539] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [15044.625669] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15054.387667] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15064.061291] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15064.187803] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [15064.281603] Lustre: lustre-OST0001: new disk, initializing [15064.286466] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [15064.297970] Lustre: 476916:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15064.386153] LNetError: 470209:0:(lib-move.c:2123:lnet_handle_find_routed_path()) peer 192.168.201.109@tcp1 has no available nets [15064.393365] LNetError: 470209:0:(lib-move.c:2123:lnet_handle_find_routed_path()) Skipped 6 previous similar messages [15064.469443] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15071.238466] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15072.469596] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15072.508382] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15072.664767] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15081.081936] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15088.109649] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15088.120810] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15088.128802] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [15088.136428] Lustre: Skipped 2 previous similar messages [15088.145962] Lustre: Skipped 1 previous similar message [15090.577168] Lustre: server umount lustre-OST0001 complete [15093.235571] LustreError: 475845:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15093.256496] LustreError: 475845:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15094.753610] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15094.757540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15094.762522] LustreError: Skipped 2 previous similar messages [15094.778135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15094.788313] Lustre: Skipped 1 previous similar message [15098.358914] LustreError: 475843:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15098.384276] LustreError: 475843:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15100.867037] Lustre: server umount lustre-OST0000 complete [15107.041025] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15107.048706] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15107.063963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15107.066672] Lustre: Skipped 3 previous similar messages [15111.679108] Lustre: server umount lustre-MDT0000 complete [15113.700832] LustreError: 474308:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15113.722230] LustreError: 474308:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15115.172767] LustreError: 474229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788166880 with bad export cookie 12081216710023956433 [15115.173879] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15115.179748] LustreError: 474229:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15115.500271] Lustre: server umount lustre-MDT0001 complete [15126.278808] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [15129.241772] Key type lgssc unregistered [15129.591751] LNet: 478888:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15129.596545] LNetError: 478888:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15129.632319] LNet: Removed LNI 192.168.201.109@tcp [15130.490729] Key type .llcrypt unregistered [15130.492709] Key type ._llcrypt unregistered [15215.235364] Key type ._llcrypt registered [15215.240582] Key type .llcrypt registered [15215.400966] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [15231.854468] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [15233.340533] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15233.412636] alg: No test for adler32 (adler32-zlib) [15234.674376] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15235.070554] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [15236.775572] Key type lgssc registered [15238.729767] Lustre: Echo OBD driver; http://www.lustre.org/ [15281.751375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15281.831563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15283.203793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15283.240346] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15283.360646] Lustre: lustre-MDT0000: new disk, initializing [15283.466482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15283.490192] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15288.519775] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15302.275650] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15302.394203] Lustre: 483825:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15302.439545] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15302.443080] Lustre: Skipped 1 previous similar message [15302.513618] Lustre: lustre-MDT0001: new disk, initializing [15302.623932] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15302.674302] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15302.696145] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15308.097603] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15319.032934] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15326.377157] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15335.565422] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15336.154062] Lustre: lustre-OST0000: new disk, initializing [15336.169027] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15336.189804] Lustre: 485795:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15336.380492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15344.988333] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15345.760304] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15345.797175] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15346.028358] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15357.639349] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15362.946285] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15363.195726] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15368.813409] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15369.068265] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15371.745596] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15371.756092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15371.784274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15371.790694] Lustre: Skipped 1 previous similar message [15376.865385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15376.877083] Lustre: Skipped 1 previous similar message [15377.788902] Lustre: server umount lustre-OST0000 complete [15384.546081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15384.562486] LustreError: Skipped 1 previous similar message [15384.570875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15384.597617] Lustre: Skipped 1 previous similar message [15384.611177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15388.577832] Lustre: server umount lustre-MDT0000 complete [15393.587062] LustreError: 483817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167159 with bad export cookie 8959468984719796255 [15393.592049] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15393.601737] LustreError: 483817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15393.912552] Lustre: server umount lustre-MDT0001 complete [15406.428444] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 05:06:09 (1788167169) [15416.382910] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 05:06:19 (1788167179) [15426.207748] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15426.642381] LustreError: 488555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15426.664821] LustreError: 488555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15426.720501] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15431.234202] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15432.168426] LustreError: 488556:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15437.283383] LustreError: 488555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15440.482410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15440.838795] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15446.125988] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15455.606498] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15462.592985] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15470.693381] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15471.138287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15478.859388] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15489.048601] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15497.751828] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [15499.881948] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [15501.818419] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [15503.973972] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [15506.110817] Lustre: Failing over lustre-MDT0000 [15506.324810] LustreError: 488551:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15506.408075] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15506.425852] Lustre: Skipped 3 previous similar messages [15506.467873] Lustre: server umount lustre-MDT0000 complete [15511.415329] LustreError: 490366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15511.439906] LustreError: 490366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15521.663441] LustreError: 490366:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15521.691764] LustreError: 490366:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [15526.003444] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15526.146246] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15526.540604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15526.603186] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [15526.764256] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [15531.937240] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15532.022984] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [15532.051305] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [15532.125904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [15542.042919] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15544.068068] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15548.346488] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [15550.523712] Lustre: Failing over lustre-MDT0000 [15550.821236] Lustre: server umount lustre-MDT0000 complete [15552.421492] LustreError: 488552:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [15552.439835] LustreError: 488552:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15552.484285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15552.488221] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15552.516628] Lustre: Skipped 1 previous similar message [15568.864428] Lustre: 482047:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788167318/real 1788167318] req@00000000722334cc x1875029000287488/t0(0) o400->MGC192.168.201.109@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788167334 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15568.894352] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15570.642832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15579.111552] LustreError: 482046:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000fd1d2e10 x1875029000293760/t0(0) o250->MGC192.168.201.109@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15579.531487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15579.590187] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [15584.113745] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [15584.731067] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15584.757870] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [15584.773921] Lustre: Skipped 2 previous similar messages [15584.820578] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [15584.881785] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [15596.324534] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15598.651509] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [15610.337043] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15610.339760] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15610.359820] Lustre: Skipped 3 previous similar messages [15610.373621] Lustre: Skipped 6 previous similar messages [15612.202316] Lustre: server umount lustre-MDT0000 complete [15615.465037] LustreError: 488555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15615.479641] LustreError: 488555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 29 previous similar messages [15616.536848] LustreError: 488536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167382 with bad export cookie 8959468984719798971 [15616.554052] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15616.557756] LustreError: 488536:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15617.014508] Lustre: server umount lustre-MDT0001 complete [15632.154746] Lustre: server umount lustre-OST0000 complete [15652.942425] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15653.844108] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15659.152282] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15670.259377] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15676.516437] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15685.074929] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15691.461303] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15699.864225] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15706.840954] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15714.861915] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15723.954798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [15734.241190] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15734.247317] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15734.253320] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15734.265120] Lustre: Skipped 1 previous similar message [15737.797946] Lustre: server umount lustre-OST0000 complete [15744.491223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15744.496412] Lustre: Skipped 1 previous similar message [15749.400142] Lustre: server umount lustre-MDT0000 complete [15749.601853] LustreError: 494895:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15749.627570] LustreError: 494895:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [15753.451803] LustreError: 494881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167519 with bad export cookie 8959468984719799713 [15753.458666] LustreError: 494881:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15753.460623] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15753.713796] Lustre: server umount lustre-MDT0001 complete [15767.679487] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [15770.485187] Key type lgssc unregistered [15770.750584] LNet: 498455:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15770.759455] LNetError: 498455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15770.779181] LNet: Removed LNI 192.168.201.109@tcp [15771.789253] Key type .llcrypt unregistered [15771.791049] Key type ._llcrypt unregistered [15798.241286] Key type ._llcrypt registered [15798.243683] Key type .llcrypt registered [15798.348125] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [15799.600619] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15799.677213] alg: No test for adler32 (adler32-zlib) [15800.726942] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [15801.002197] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [15802.639194] Key type lgssc registered [15803.742980] Lustre: Echo OBD driver; http://www.lustre.org/ [15811.504057] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15811.523268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15812.970569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15818.457554] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15823.336944] LustreError: 499972:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15823.354425] LustreError: 499972:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15826.646687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15827.119652] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15831.678267] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15839.641769] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15845.893596] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15852.308971] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15852.861571] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15857.973178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [15858.817434] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15866.415904] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15887.848687] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15887.874666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15887.880907] Lustre: Skipped 2 previous similar messages [15887.896510] Lustre: Skipped 2 previous similar messages [15890.609475] Lustre: server umount lustre-MDT0000 complete [15892.962921] LustreError: 499968:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15892.975249] LustreError: 499968:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [15895.974786] LustreError: 499954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167661 with bad export cookie 8786786453700071205 [15895.985576] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15896.003197] LustreError: 499954:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [15896.779784] Lustre: server umount lustre-MDT0001 complete [15910.712467] Lustre: server umount lustre-OST0000 complete [15924.559614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15925.015737] LustreError: 503704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15925.091252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15929.872801] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15930.339851] LustreError: 503705:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15934.432614] LustreError: 503704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15939.800934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15940.124496] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15944.768246] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15954.343716] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15961.287099] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15968.546968] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15968.935918] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15969.985604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [15975.079761] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [15982.123375] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15995.878810] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15995.893179] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15995.896763] Lustre: Skipped 1 previous similar message [15995.915240] Lustre: Skipped 2 previous similar messages [15999.813606] Lustre: server umount lustre-MDT0000 complete [16001.001384] LustreError: 503699:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16001.022474] LustreError: 503699:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16004.136199] LustreError: 503684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167769 with bad export cookie 8786786453700072318 [16004.141339] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16004.145724] LustreError: 503684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [16004.460676] Lustre: server umount lustre-MDT0001 complete [16017.721495] Lustre: server umount lustre-OST0000 complete [16038.225735] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16039.042347] LustreError: 507615:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16039.138772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16044.758897] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16055.697900] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16061.145530] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16069.743600] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16077.078559] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16085.127567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16085.486762] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16085.495916] Lustre: Skipped 1 previous similar message [16092.218646] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16100.491548] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16109.046382] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [16119.266764] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16119.269060] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16119.274270] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16119.310075] Lustre: Skipped 2 previous similar messages [16119.334370] Lustre: Skipped 1 previous similar message [16124.515069] Lustre: server umount lustre-OST0000 complete [16129.505633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16129.509193] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16129.521717] Lustre: Skipped 1 previous similar message [16129.534053] Lustre: Skipped 1 previous similar message [16134.627353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16134.637599] LustreError: 507611:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16134.654764] LustreError: 507611:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16134.771812] Lustre: server umount lustre-MDT0000 complete [16140.178900] LustreError: 507597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788167905 with bad export cookie 8786786453700073340 [16140.212237] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16140.213130] LustreError: 507597:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16140.575276] Lustre: server umount lustre-MDT0001 complete [16158.373608] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [16163.108316] Key type lgssc unregistered [16163.436673] LNet: 511191:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16163.460329] LNetError: 511191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16164.532702] LNet: Removed LNI 192.168.201.109@tcp [16165.762286] Key type .llcrypt unregistered [16165.765543] Key type ._llcrypt unregistered [16199.846318] Key type ._llcrypt registered [16199.848759] Key type .llcrypt registered [16199.987685] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [16201.552996] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16201.635188] alg: No test for adler32 (adler32-zlib) [16202.977272] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16203.331966] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [16205.119281] Key type lgssc registered [16206.388207] Lustre: Echo OBD driver; http://www.lustre.org/ [16214.435620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16214.454831] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16214.460376] LustreError: 512633:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [16214.473084] LustreError: 512633:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [16214.480633] LustreError: 512633:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [16214.490322] LustreError: 512633:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [16214.495135] LustreError: 512633:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [16223.147837] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [16223.156555] LustreError: 513030:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [16223.175970] LustreError: 513030:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [16223.180897] LustreError: 513030:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [16223.187611] LustreError: 513030:0:(tgt_mount.c:2656:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [16223.194295] LustreError: 513030:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [16245.305758] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [16248.283543] Key type lgssc unregistered [16248.647065] LNet: 514033:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16248.662924] LNetError: 514033:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16248.692119] LNet: Removed LNI 192.168.201.109@tcp [16249.782323] Key type .llcrypt unregistered [16249.785314] Key type ._llcrypt unregistered [16268.685918] Key type ._llcrypt registered [16268.687585] Key type .llcrypt registered [16268.765062] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [16285.742213] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [16287.308931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16287.429565] alg: No test for adler32 (adler32-zlib) [16288.573778] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16288.924461] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [16290.639197] Key type lgssc registered [16292.206611] Lustre: Echo OBD driver; http://www.lustre.org/ [16338.574163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16338.622678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16340.099910] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16340.149073] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16340.226601] Lustre: lustre-MDT0000: new disk, initializing [16340.297989] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16340.318300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16345.718969] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16357.547843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16357.794884] Lustre: 517192:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16357.899069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16357.909514] Lustre: Skipped 1 previous similar message [16358.012297] Lustre: lustre-MDT0001: new disk, initializing [16358.148399] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16358.201145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16358.217184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16363.835816] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16375.529377] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16382.624891] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16390.902893] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16391.217063] Lustre: lustre-OST0000: new disk, initializing [16391.221636] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16391.234508] Lustre: 519162:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16391.365683] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16398.530200] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16399.444856] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16399.449683] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16399.616805] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16409.747612] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16415.134824] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16415.340043] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16420.241245] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16420.490154] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16425.445403] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16425.448217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16425.456103] LustreError: Skipped 1 previous similar message [16425.475515] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16425.504145] Lustre: Skipped 1 previous similar message [16430.563588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16430.570623] Lustre: Skipped 1 previous similar message [16435.695499] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16435.709291] Lustre: Skipped 1 previous similar message [16437.215145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16437.413845] Lustre: server umount lustre-OST0000 complete [16445.409749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16445.419695] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16445.439271] Lustre: Skipped 1 previous similar message [16445.447953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16448.751594] Lustre: server umount lustre-MDT0000 complete [16452.784950] LustreError: 517184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168218 with bad export cookie 5370165520128834111 [16452.802632] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16453.267356] Lustre: server umount lustre-MDT0001 complete [16465.649302] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [16468.208132] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 05:23:50 (1788168230) [16475.377606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16475.907826] LustreError: 521627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16475.915727] LustreError: 521627:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16476.026428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16480.818156] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16481.265732] LustreError: 521628:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16486.386454] LustreError: 521627:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16490.672898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16491.052728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16497.065836] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16507.118866] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16514.890775] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16523.120852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16523.578197] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16530.909539] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16539.272960] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16547.849844] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16548.178888] Lustre: lustre-OST0001: new disk, initializing [16548.182555] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [16548.189623] Lustre: 524465:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16548.319084] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [16555.641188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [16555.660408] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [16555.848289] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [16556.589697] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16569.141852] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16576.452881] Lustre: lustre-OST0001: Not available for connect from 192.168.201.9@tcp (stopping) [16576.471106] Lustre: Skipped 1 previous similar message [16576.494212] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16576.501167] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16576.510791] LustreError: Skipped 1 previous similar message [16576.559787] Lustre: Skipped 2 previous similar messages [16581.492246] LustreError: 523451:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16581.559311] Lustre: server umount lustre-OST0001 complete [16586.609491] LustreError: 523452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [16586.628024] LustreError: 523452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16596.993783] LustreError: 524783:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16597.014155] LustreError: 524783:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16602.080781] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16602.093468] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16602.104991] Lustre: Skipped 1 previous similar message [16602.110435] Lustre: Skipped 2 previous similar messages [16608.252683] Lustre: server umount lustre-OST0000 complete [16613.866427] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16613.883436] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16618.785717] Lustre: server umount lustre-MDT0000 complete [16622.561718] LustreError: 521622:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16622.586152] LustreError: 521622:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16623.367029] LustreError: 521607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168388 with bad export cookie 5370165520128834993 [16623.378981] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16623.388158] LustreError: 521607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [16629.876309] Lustre: server umount lustre-MDT0001 complete [16643.926903] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [16646.911733] Key type lgssc unregistered [16647.288330] LNet: 526455:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16647.304298] LNetError: 526455:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16647.336687] LNet: Removed LNI 192.168.201.109@tcp [16648.627685] Key type .llcrypt unregistered [16648.631405] Key type ._llcrypt unregistered [16706.684780] Key type ._llcrypt registered [16706.688650] Key type .llcrypt registered [16706.803832] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [16708.192353] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16708.341901] alg: No test for adler32 (adler32-zlib) [16709.383041] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [16709.621641] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [16711.335849] Key type lgssc registered [16712.629675] Lustre: Echo OBD driver; http://www.lustre.org/ [16720.218202] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16720.238392] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16721.455529] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [16721.469557] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16721.614540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16726.509268] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16737.047746] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16737.257238] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [16737.306711] Lustre: 529256:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16737.681321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16743.942949] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16755.205158] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16762.167555] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16768.480394] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16768.482699] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16768.489143] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16768.501024] Lustre: Skipped 1 previous similar message [16768.519980] Lustre: Skipped 1 previous similar message [16770.915371] Lustre: server umount lustre-MDT0000 complete [16775.210508] LustreError: 529247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168540 with bad export cookie 11481044043009998883 [16775.222116] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16775.236979] LustreError: 529247:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16775.645267] Lustre: server umount lustre-MDT0001 complete [16786.421338] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 05:29:09 (1788168549) [16793.826034] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16794.228873] LustreError: 531768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16794.246723] LustreError: 531768:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16794.293163] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16799.008637] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16799.719215] LustreError: 531769:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16804.832588] LustreError: 531768:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16808.309661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16808.618031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16812.970536] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16820.546905] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16826.950811] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16834.311567] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16834.487890] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [16834.629781] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16841.440720] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16849.573992] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16858.539513] Lustre: Permanently deactivating lustre-OST0000 [16858.555279] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [16866.383532] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16869.554106] LustreError: 534758:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [16869.564523] LustreError: 534758:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [16869.573211] LustreError: 534758:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [16872.580883] LustreError: 534806:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [16872.587492] LustreError: 534806:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [16881.387248] Lustre: server umount lustre-OST0000 complete [16885.212798] LustreError: 535056:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [16885.220634] LustreError: 535056:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [16890.851650] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16890.854506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16890.883231] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16896.491547] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16896.504189] Lustre: Skipped 3 previous similar messages [16897.593278] Lustre: server umount lustre-MDT0000 complete [16901.389289] LustreError: 531750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168667 with bad export cookie 11481044043009999471 [16901.391285] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16901.401586] LustreError: 531750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16911.986622] Lustre: server umount lustre-MDT0001 complete [16922.381662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16927.943127] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16930.411552] LustreError: 536217:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [16938.294741] Lustre: 536557:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.109@tcp [16941.233893] Lustre: server umount MGS complete [16954.013718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16954.334794] LustreError: 537281:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16954.346491] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16954.350818] Lustre: Skipped 1 previous similar message [16954.388757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16959.265368] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16959.459228] LustreError: 537282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16967.258504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16967.462900] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [16972.081421] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [16979.573071] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16986.023138] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16988.153776] Lustre: Permanently reactivating lustre-OST0000 [16988.199672] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [16988.208657] Lustre: Skipped 2 previous similar messages [17016.799764] LustreError: 539067:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [17025.385900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17025.775128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17025.780426] Lustre: Skipped 1 previous similar message [17033.065502] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17041.680405] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17047.523782] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17047.524161] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17047.548741] Lustre: Skipped 1 previous similar message [17047.556724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17047.572787] Lustre: Skipped 1 previous similar message [17049.436391] Lustre: server umount lustre-OST0000 complete [17054.688533] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17054.696067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17054.712915] Lustre: Skipped 1 previous similar message [17054.733642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17059.585888] Lustre: server umount lustre-MDT0000 complete [17062.880448] LustreError: 537278:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17062.904538] LustreError: 537278:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17063.324942] LustreError: 537261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788168828 with bad export cookie 11481044043010000521 [17063.330384] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17063.334612] LustreError: 537261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [17063.701482] Lustre: server umount lustre-MDT0001 complete [17074.534714] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [17077.843247] Key type lgssc unregistered [17078.136326] LNet: 541691:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17078.154484] LNetError: 541691:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17078.202945] LNet: Removed LNI 192.168.201.109@tcp [17079.060336] Key type .llcrypt unregistered [17079.062801] Key type ._llcrypt unregistered [17100.260075] Key type ._llcrypt registered [17100.261571] Key type .llcrypt registered [17100.400655] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [17120.844436] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [17123.348145] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17123.400196] alg: No test for adler32 (adler32-zlib) [17124.553824] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17124.762455] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [17126.455196] Key type lgssc registered [17128.162587] Lustre: Echo OBD driver; http://www.lustre.org/ [17165.305283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17165.325489] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17166.529311] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17166.572975] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17166.694724] Lustre: lustre-MDT0000: new disk, initializing [17166.793797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17166.837384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17171.412465] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17181.766352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17181.923257] Lustre: 544851:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17181.963968] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17181.968352] Lustre: Skipped 1 previous similar message [17182.055906] Lustre: lustre-MDT0001: new disk, initializing [17182.143503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17182.210689] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17182.224506] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17186.499399] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17196.708869] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17203.403202] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17211.120280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17211.494818] Lustre: lustre-OST0000: new disk, initializing [17211.500254] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17211.506861] Lustre: 546822:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17211.597185] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17220.462569] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17221.237496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17221.247128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17221.381654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17231.660968] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17237.135252] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17237.367912] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17241.990742] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17242.198403] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17247.205517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17247.207177] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17247.215787] LustreError: Skipped 1 previous similar message [17247.232609] Lustre: Skipped 1 previous similar message [17247.237419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17247.250485] Lustre: Skipped 1 previous similar message [17250.520713] Lustre: server umount lustre-OST0000 complete [17257.441653] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17257.459774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17261.427477] Lustre: server umount lustre-MDT0000 complete [17265.323650] LustreError: 545649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169030 with bad export cookie 12165443125690779140 [17265.327641] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17265.340880] LustreError: 545649:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17265.765836] Lustre: server umount lustre-MDT0001 complete [17276.511652] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [17278.265992] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 05:37:21 (1788169041) [17285.467110] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17285.890505] LustreError: 549282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17285.905498] LustreError: 549282:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [17285.944349] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17290.589913] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17291.233661] LustreError: 549283:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17296.356149] LustreError: 549282:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17299.989852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17300.474430] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17305.331885] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17313.218723] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17319.612587] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17326.473897] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17326.730545] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17333.302568] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17343.303098] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17345.681338] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [17348.868809] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [17356.768808] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17356.777044] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17356.789243] LustreError: Skipped 1 previous similar message [17356.798654] Lustre: Skipped 1 previous similar message [17356.804296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17356.804308] Lustre: Skipped 1 previous similar message [17360.569697] Lustre: server umount lustre-OST0000 complete [17367.008385] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17367.011741] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17367.018100] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17367.046100] Lustre: Skipped 2 previous similar messages [17367.070288] Lustre: Skipped 2 previous similar messages [17372.131038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17372.142352] Lustre: Skipped 1 previous similar message [17380.319161] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17380.665824] Lustre: server umount lustre-MDT0000 complete [17382.372156] LustreError: 550488:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17382.393934] LustreError: 550488:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17385.206458] LustreError: 549264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169150 with bad export cookie 12165443125690780022 [17385.208687] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17385.228103] LustreError: 549264:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17385.709888] Lustre: server umount lustre-MDT0001 complete [17397.734653] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [17400.521538] Key type lgssc unregistered [17400.776836] LNet: 553010:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17400.782409] LNetError: 553010:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17400.800650] LNet: Removed LNI 192.168.201.109@tcp [17401.866340] Key type .llcrypt unregistered [17401.870128] Key type ._llcrypt unregistered [17428.334190] Key type ._llcrypt registered [17428.337969] Key type .llcrypt registered [17428.412581] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [17428.655647] Key type .llcrypt unregistered [17428.658418] Key type ._llcrypt unregistered [17446.533194] Key type ._llcrypt registered [17446.534832] Key type .llcrypt registered [17446.659437] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [17447.867910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17447.956259] alg: No test for adler32 (adler32-zlib) [17448.949668] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17449.157905] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [17450.879649] Key type lgssc registered [17451.914978] Lustre: Echo OBD driver; http://www.lustre.org/ [17460.153982] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17460.172549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17461.593070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17466.211701] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17471.970817] LustreError: 555223:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17471.985549] LustreError: 555223:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17477.092612] LustreError: 555224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17477.306075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17477.755500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17483.238297] LustreError: 555219:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17483.259691] LustreError: 555219:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17484.044957] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17487.353442] LustreError: 555224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17487.375445] LustreError: 555224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17487.377142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [17492.455498] LustreError: 555218:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17492.474495] LustreError: 555218:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17492.832453] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17499.741940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17500.029946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17503.772491] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17512.936444] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17512.950130] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17512.979225] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17513.545777] Lustre: server umount lustre-OST0000 complete [17518.854596] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [17518.857255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17518.869825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17519.100231] Lustre: server umount lustre-MDT0000 complete [17520.611950] LustreError: 555224:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17520.614463] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17520.618957] LustreError: 555224:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17523.763350] LustreError: 555202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169289 with bad export cookie 13271684427767917707 [17523.791637] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17523.807184] LustreError: 555202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17533.590315] Lustre: server umount lustre-MDT0001 complete [17546.366313] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [17549.018260] Key type lgssc unregistered [17549.281374] LNet: 558375:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17549.304405] LNetError: 558375:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17550.375303] LNet: Removed LNI 192.168.201.109@tcp [17551.178429] Key type .llcrypt unregistered [17551.184584] Key type ._llcrypt unregistered [17575.913982] Key type ._llcrypt registered [17575.915861] Key type .llcrypt registered [17576.002655] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [17577.007742] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17577.111722] alg: No test for adler32 (adler32-zlib) [17578.142079] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17578.412860] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [17580.175237] Key type lgssc registered [17581.633166] Lustre: Echo OBD driver; http://www.lustre.org/ [17590.744706] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17590.773394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17606.431636] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [17611.551526] LustreError: 559768:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.109@tcp: failed processing log, type 4: rc = -110 [17638.239524] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17645.009928] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17653.705414] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17661.192102] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17661.937307] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17662.066605] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [17667.044959] LustreError: 559793:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17667.064322] LustreError: 559793:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17667.720086] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17672.166404] LustreError: 559794:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17673.192862] LustreError: 560804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17673.222394] LustreError: 560804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17677.949915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17678.393800] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17683.302924] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17693.993483] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17702.847257] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17713.120901] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17713.128167] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17713.162924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17714.157079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17714.157263] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17714.182412] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17714.188678] Lustre: Skipped 1 previous similar message [17716.837289] Lustre: server umount lustre-OST0000 complete [17724.384734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17724.389636] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17724.403394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17724.409908] Lustre: Skipped 1 previous similar message [17727.211951] Lustre: server umount lustre-MDT0000 complete [17729.507295] LustreError: 560804:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17729.539478] LustreError: 560804:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17731.072265] LustreError: 559775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169496 with bad export cookie 9740140434585785025 [17731.076993] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17731.084522] LustreError: 559775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17731.479313] Lustre: server umount lustre-MDT0001 complete [17744.380857] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [17747.526519] Key type lgssc unregistered [17747.855183] LNet: 563338:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17747.881363] LNetError: 563338:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17747.902422] LNet: Removed LNI 192.168.201.109@tcp [17748.811784] Key type .llcrypt unregistered [17748.813513] Key type ._llcrypt unregistered [17781.810765] Key type ._llcrypt registered [17781.819377] Key type .llcrypt registered [17781.958749] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [17783.778315] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17784.024608] alg: No test for adler32 (adler32-zlib) [17785.421549] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17786.028778] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [17787.991345] Key type lgssc registered [17790.231104] Lustre: Echo OBD driver; http://www.lustre.org/ [17800.901563] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17800.948752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17802.744783] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17808.504577] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17812.981361] LustreError: 564757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17813.005288] LustreError: 564757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17818.081046] LustreError: 564758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17820.275231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17820.830362] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17827.510378] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17835.741712] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17841.464868] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17849.388432] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17849.691942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17856.612388] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17865.735748] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17873.219591] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [17882.082337] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [17882.102725] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17882.152643] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17883.038511] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.9@tcp (stopping) [17883.634170] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17883.651347] Lustre: Skipped 1 previous similar message [17887.215469] Lustre: server umount lustre-MDT0000 complete [17888.117656] LustreError: 565896:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [17889.955113] LustreError: 564739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.9@tcp arrived at 1788169655 with bad export cookie 3279389063192548826 [17889.981604] LustreError: 564739:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [17892.209249] LustreError: 564738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788169657 with bad export cookie 3279389063192548161 [17892.239887] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17892.834981] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [17892.851137] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17892.874382] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17892.888020] Lustre: Skipped 3 previous similar messages [17893.858875] LustreError: 564757:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17893.901230] LustreError: 564757:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [17897.895298] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17897.896448] LustreError: 564758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17897.904908] Lustre: Skipped 1 previous similar message [17897.941519] LustreError: 564758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17898.712165] Lustre: server umount lustre-OST0000 complete [17907.175617] LustreError: 564758:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17907.211906] LustreError: 564758:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [17908.571668] Lustre: server umount lustre-MDT0001 complete [17922.302175] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [17925.214454] Key type lgssc unregistered [17925.543786] LNet: 568465:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17925.550444] LNetError: 568465:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17925.569315] LNet: Removed LNI 192.168.201.109@tcp [17926.445235] Key type .llcrypt unregistered [17926.452221] Key type ._llcrypt unregistered [17954.874047] Key type ._llcrypt registered [17954.877166] Key type .llcrypt registered [17955.050363] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [17956.621875] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17956.670796] alg: No test for adler32 (adler32-zlib) [17957.703808] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [17958.072782] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [17959.887139] Key type lgssc registered [17961.566052] Lustre: Echo OBD driver; http://www.lustre.org/ [17971.243558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17971.270579] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17972.700550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17977.439031] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17982.953395] LustreError: 569883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17982.969747] LustreError: 569883:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [17985.543343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17986.029408] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17990.434109] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [17999.124148] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18006.245917] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18014.014674] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18014.380109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18017.474293] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [18022.143153] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [18031.959885] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18040.204250] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [18041.321121] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18041.334520] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [18041.344872] Lustre: Skipped 1 previous similar message [18043.784367] Lustre: server umount lustre-MDT0001 complete [18045.295928] LustreError: 569879:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.9@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [18046.437364] LustreError: 569884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18046.452078] LustreError: 569884:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18051.554604] LustreError: 569884:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18051.555763] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18051.583942] Lustre: Skipped 1 previous similar message [18051.595919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18056.677632] LustreError: 569883:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18056.685620] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18057.059547] Lustre: server umount lustre-OST0000 complete [18061.781895] Lustre: server umount lustre-MDT0000 complete [18076.793389] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [18079.938187] Key type lgssc unregistered [18080.231064] LNet: 573565:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18080.238810] LNetError: 573565:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18080.263952] LNet: Removed LNI 192.168.201.109@tcp [18081.498213] Key type .llcrypt unregistered [18081.501637] Key type ._llcrypt unregistered [18110.504596] Key type ._llcrypt registered [18110.506691] Key type .llcrypt registered [18110.588243] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [18126.266275] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [18127.232065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18127.275692] alg: No test for adler32 (adler32-zlib) [18128.334731] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [18128.602509] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [18130.271199] Key type lgssc registered [18131.700786] Lustre: Echo OBD driver; http://www.lustre.org/ [18174.789811] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [18192.945253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18192.982606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18194.231189] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18194.260985] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18194.361965] Lustre: lustre-MDT0000: new disk, initializing [18194.434739] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18194.511321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18198.779464] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [18215.327315] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18215.821977] Lustre: lustre-OST0000: new disk, initializing [18215.826657] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18215.833831] Lustre: Skipped 1 previous similar message [18215.838612] Lustre: 578605:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18215.959371] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18223.228415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [18223.241177] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [18223.289748] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [18223.467797] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [18236.143389] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18254.867412] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18255.002735] Lustre: 577586:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18255.054991] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18255.139175] Lustre: lustre-MDT0001: new disk, initializing [18255.266682] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18255.305588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [18255.314129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [18260.476492] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [18272.105107] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [18272.403488] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [18277.854929] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [18278.129908] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [18283.913680] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [18284.006336] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18284.007427] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [18289.015204] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.9@tcp (stopping) [18289.023372] Lustre: Skipped 2 previous similar messages [18289.867962] Lustre: server umount lustre-MDT0001 complete [18299.361177] LustreError: 577600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18299.389925] LustreError: 577600:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [18304.483761] LustreError: 577595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18304.501083] LustreError: 577595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [18309.602112] LustreError: 577599:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18309.604359] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18309.656563] Lustre: Skipped 1 previous similar message [18309.659058] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18309.669539] Lustre: Skipped 2 previous similar messages [18314.557753] Lustre: server umount lustre-OST0000 complete [18314.737920] LustreError: 577600:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18325.008459] Lustre: server umount lustre-MDT0000 complete [18340.213648] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [18342.999823] Key type lgssc unregistered [18343.288335] LNet: 582355:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18343.295416] LNetError: 582355:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18343.321084] LNet: Removed LNI 192.168.201.109@tcp [18344.292124] Key type .llcrypt unregistered [18344.293570] Key type ._llcrypt unregistered [18389.614490] Key type ._llcrypt registered [18389.615597] Key type .llcrypt registered [18389.684640] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [18390.009500] Key type .llcrypt unregistered [18390.013764] Key type ._llcrypt unregistered [18410.826965] Key type ._llcrypt registered [18410.830794] Key type .llcrypt registered [18411.028169] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [18412.276783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18412.390248] alg: No test for adler32 (adler32-zlib) [18413.648629] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [18413.893953] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [18415.543561] Key type lgssc registered [18416.752132] Lustre: Echo OBD driver; http://www.lustre.org/ [18448.246547] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18448.290468] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18449.697512] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18449.775946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18450.002662] Lustre: lustre-MDT0000: new disk, initializing [18450.156174] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18450.190656] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18456.076481] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [18468.364410] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18468.536648] Lustre: 585737:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18468.647634] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18468.653043] Lustre: Skipped 1 previous similar message [18468.786867] Lustre: lustre-MDT0001: new disk, initializing [18468.940247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18468.979979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18469.007946] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18474.275122] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [18486.349983] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18492.598299] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18500.141211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18500.515815] Lustre: lustre-OST0000: new disk, initializing [18500.519187] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18500.524418] Lustre: 587706:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18500.653739] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18506.658751] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [18507.340885] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18507.354535] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18507.448411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18516.028651] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18525.153415] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [18525.162134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18525.175847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18528.231451] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18528.232904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18528.257912] Lustre: Skipped 1 previous similar message [18528.279257] Lustre: Skipped 2 previous similar messages [18530.693802] Lustre: server umount lustre-MDT0000 complete [18534.557870] LustreError: 585727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170300 with bad export cookie 17871754430516189634 [18534.573846] LustreError: 585727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [18534.576705] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18534.946682] Lustre: server umount lustre-MDT0001 complete [18549.153597] Lustre: server umount lustre-OST0000 complete [18562.415600] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 05:58:45 (1788170325) [18572.165157] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18572.770726] LustreError: 589734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18572.802043] LustreError: 589734:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [18572.844013] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18577.907213] LustreError: 589735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18579.825819] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [18583.012943] LustreError: 589734:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18588.129584] LustreError: 589735:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18589.557948] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18590.173358] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18594.583918] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [18602.105706] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18608.020704] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18627.764410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18628.009416] Lustre: Found index 0 for lustre-OST0000, updating log [18628.167505] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18634.844856] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [18642.769888] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18644.594702] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [18655.203801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18655.215406] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [18655.263513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18662.373090] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18662.380945] Lustre: Skipped 3 previous similar messages [18666.975146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [18667.205240] Lustre: server umount lustre-OST0000 complete [18683.872468] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18683.885795] Lustre: Skipped 1 previous similar message [18683.889054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18683.891500] Lustre: Skipped 1 previous similar message [18689.375447] LNet: 1 local NIs in recovery (showing 1): 192.168.201.109@tcp [18689.657726] Lustre: server umount lustre-MDT0000 complete [18693.852619] LustreError: 589714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788170459 with bad export cookie 17871754430516191027 [18693.859778] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18693.873155] LustreError: 589714:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [18694.678212] Lustre: server umount lustre-MDT0001 complete [18706.258475] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 06:01:09 (1788170469) [19339.115073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19339.781929] LustreError: 596159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19339.916945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19345.260834] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [19345.382416] LustreError: 596160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19349.475444] LustreError: 596159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19354.997933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19355.460816] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19360.619777] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [19370.078637] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19377.635818] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19394.911551] LNet: 1 local NIs in recovery (showing 1): 192.168.201.109@tcp [19399.098152] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19399.534456] Lustre: Found index 0 for lustre-OST0000, updating log [19399.993902] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19408.949704] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [19417.259208] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19419.625675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [19419.637267] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19419.653336] Lustre: Skipped 1 previous similar message [19419.667832] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19419.680747] Lustre: Skipped 3 previous similar messages [19425.500652] Lustre: server umount lustre-OST0000 complete [19431.085548] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19431.095320] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19431.100470] Lustre: Skipped 1 previous similar message [19431.114575] Lustre: Skipped 1 previous similar message [19433.189221] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19433.198969] Lustre: Skipped 3 previous similar messages [19436.294741] Lustre: server umount lustre-MDT0000 complete [19436.332266] LustreError: 596155:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19436.356170] LustreError: 596155:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [19440.489502] LustreError: 596160:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19440.507490] LustreError: 596160:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [19440.563901] LustreError: 596139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171206 with bad export cookie 17871754430516192105 [19440.570849] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19440.579035] LustreError: 596139:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [19441.029967] Lustre: server umount lustre-MDT0001 complete [19455.327201] LNet: 1 local NIs in recovery (showing 1): 192.168.201.109@tcp [19476.475531] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [19489.958117] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [19532.898872] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 06:14:55 (1788171295) [19553.467690] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [19556.476574] Key type lgssc unregistered [19556.852781] LNet: 602773:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19556.871390] LNetError: 602773:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19662.386744] LNet: Removed LNI 192.168.201.109@tcp [19664.223793] Key type .llcrypt unregistered [19664.232119] Key type ._llcrypt unregistered [19682.001867] Key type ._llcrypt registered [19682.008275] Key type .llcrypt registered [19682.132691] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [19683.407135] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19683.561371] alg: No test for adler32 (adler32-zlib) [19684.746750] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19684.973990] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [19686.647188] Key type lgssc registered [19687.977284] Lustre: Echo OBD driver; http://www.lustre.org/ [19693.614679] LNet: Added LNI 192.168.201.109@tcp100 [8/256/0/180] [19694.882502] LNet: Added LNI 192.168.201.109@tcp101 [8/256/0/180] [19696.179241] LNet: Added LNI 192.168.201.109@tcp102 [8/256/0/180] [19697.298895] LNet: Added LNI 192.168.201.109@tcp103 [8/256/0/180] [19698.470074] LNet: Added LNI 192.168.201.109@tcp104 [8/256/0/180] [19699.309940] LNet: Added LNI 192.168.201.109@tcp105 [8/256/0/180] [19700.522540] LNet: Added LNI 192.168.201.109@tcp106 [8/256/0/180] [19701.460283] LNet: Added LNI 192.168.201.109@tcp107 [8/256/0/180] [19702.465697] LNet: Added LNI 192.168.201.109@tcp108 [8/256/0/180] [19703.613621] LNet: Added LNI 192.168.201.109@tcp109 [8/256/0/180] [19704.521218] LNet: Added LNI 192.168.201.109@tcp110 [8/256/0/180] [19705.631733] LNet: Added LNI 192.168.201.109@tcp111 [8/256/0/180] [19706.664635] LNet: Added LNI 192.168.201.109@tcp112 [8/256/0/180] [19707.657915] LNet: Added LNI 192.168.201.109@tcp113 [8/256/0/180] [19708.740816] LNet: Added LNI 192.168.201.109@tcp114 [8/256/0/180] [19709.754279] LNet: Added LNI 192.168.201.109@tcp115 [8/256/0/180] [19710.741863] LNet: Added LNI 192.168.201.109@tcp116 [8/256/0/180] [19711.533133] LNet: Added LNI 192.168.201.109@tcp117 [8/256/0/180] [19712.486873] LNet: Added LNI 192.168.201.109@tcp118 [8/256/0/180] [19713.576864] LNet: Added LNI 192.168.201.109@tcp119 [8/256/0/180] [19714.562715] LNet: Added LNI 192.168.201.109@tcp120 [8/256/0/180] [19715.796400] LNet: Added LNI 192.168.201.109@tcp121 [8/256/0/180] [19716.821354] LNet: Added LNI 192.168.201.109@tcp122 [8/256/0/180] [19717.840151] LNet: Added LNI 192.168.201.109@tcp123 [8/256/0/180] [19718.907801] LNet: Added LNI 192.168.201.109@tcp124 [8/256/0/180] [19719.871389] LNet: Added LNI 192.168.201.109@tcp125 [8/256/0/180] [19720.997313] LNet: Added LNI 192.168.201.109@tcp126 [8/256/0/180] [19722.053699] LNet: Added LNI 192.168.201.109@tcp127 [8/256/0/180] [19723.106614] LNet: Added LNI 192.168.201.109@tcp128 [8/256/0/180] [19724.153741] LNet: Added LNI 192.168.201.109@tcp129 [8/256/0/180] [19725.086858] LNet: Added LNI 192.168.201.109@tcp130 [8/256/0/180] [19726.292391] LNet: Added LNI 192.168.201.109@tcp131 [8/256/0/180] [19727.043925] LNet: Added LNI 192.168.201.109@tcp132 [8/256/0/180] [19728.063452] LNet: Added LNI 192.168.201.109@tcp133 [8/256/0/180] [19729.000524] LNet: Added LNI 192.168.201.109@tcp134 [8/256/0/180] [19730.075778] LNet: Added LNI 192.168.201.109@tcp135 [8/256/0/180] [19737.545552] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19737.581791] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19738.722030] Lustre: 605818:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [19738.797428] LNetError: 605841:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [19738.805478] LNetError: Protocol error connecting to 192.168.201.109@tcp122 at host 192.168.201.109:988: is it running a compatible version of Lustre? [19738.817861] LNetError: Skipped 4 previous similar messages [19738.942748] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19738.975814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19739.103195] Lustre: lustre-MDT0000: new disk, initializing [19739.198579] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19739.216121] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19739.231169] LNet: 1 local NIs in recovery (showing 1): 192.168.201.109@tcp122 [19741.280552] LNetError: 605838:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [19741.286548] LNetError: 605838:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 12 previous similar messages [19741.301802] LNetError: Protocol error connecting to 192.168.201.109@tcp122 at host 192.168.201.109:988: is it running a compatible version of Lustre? [19741.309945] LNetError: Skipped 8 previous similar messages [19743.744742] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [19745.387510] LNetError: 605843:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [19745.405700] LNetError: Protocol error connecting to 192.168.201.109@tcp122 at host 192.168.201.109:988: is it running a compatible version of Lustre? [19753.568667] LNetError: 605848:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [19753.579560] LNetError: Protocol error connecting to 192.168.201.109@tcp122 at host 192.168.201.109:988: is it running a compatible version of Lustre? [19754.156193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19754.264651] Lustre: 605834:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19754.327558] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19754.331753] Lustre: Skipped 1 previous similar message [19754.392730] Lustre: lustre-MDT0001: new disk, initializing [19754.425171] Lustre: 606635:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [19754.429431] Lustre: 606635:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [19754.533423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19754.576730] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19754.593303] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19760.036637] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [19769.963637] LNetError: 603274:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [19769.986188] LNetError: Protocol error connecting to 192.168.201.109@tcp122 at host 192.168.201.109:988: is it running a compatible version of Lustre? [19771.475861] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19778.211709] LNetError: 605848:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [19778.248960] LNetError: 605848:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 8 previous similar messages [19778.270577] LNetError: Protocol error connecting to 192.168.201.109@tcp107 at host 192.168.201.109:988: is it running a compatible version of Lustre? [19778.290135] LNetError: Skipped 8 previous similar messages [19779.304601] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19786.883574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19787.095633] Lustre: lustre-OST0000: new disk, initializing [19787.098666] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19787.103522] Lustre: 607808:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19787.204916] Lustre: 607813:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [19787.211632] Lustre: 607813:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [19787.277568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19794.597088] LNetError: 603276:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [19794.615019] LNetError: 603276:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 13 previous similar messages [19794.623028] LNetError: Protocol error connecting to 192.168.201.109@tcp105 at host 192.168.201.109:988: is it running a compatible version of Lustre? [19794.638665] LNetError: Skipped 13 previous similar messages [19795.025963] Lustre: 608120:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.109@tcp: can't add NID 192.168.201.109@tcp130: rc = -75 [19795.032157] Lustre: 608120:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [19795.089117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19795.112319] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19795.303841] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19795.669442] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [19798.623171] LNet: 1 local NIs in recovery (showing 1): 192.168.201.109@tcp122 [19807.154198] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19821.024504] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [19821.026476] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19821.042305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19821.079424] Lustre: Skipped 1 previous similar message [19823.656856] Lustre: server umount lustre-OST0000 complete [19827.426974] LNetError: 603275:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.109 [19827.447190] LNetError: 603275:0:(socklnd_cb.c:1686:ksocknal_recv_hello()) Skipped 19 previous similar messages [19827.452032] LNetError: Protocol error connecting to 192.168.201.109@tcp117 at host 192.168.201.109:988: is it running a compatible version of Lustre? [19827.474557] LNetError: Skipped 19 previous similar messages [19831.269249] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19831.275184] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19831.299832] LustreError: Skipped 1 previous similar message [19831.345198] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19831.368688] Lustre: Skipped 2 previous similar messages [19835.187804] Lustre: server umount lustre-MDT0000 complete [19839.809142] LustreError: 605827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788171605 with bad export cookie 3969249479775045921 [19839.813782] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19839.823513] LustreError: 605827:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [19840.188917] Lustre: server umount lustre-MDT0001 complete [19852.562805] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [19855.869066] Key type lgssc unregistered [19856.253111] LNet: 609724:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19856.262546] LNetError: 609724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19857.311237] LNet: Removed LNI 192.168.201.109@tcp [19857.321846] LNetError: 609724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19858.335252] LNet: Removed LNI 192.168.201.109@tcp100 [19858.344973] LNetError: 609724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19858.370661] LNet: Removed LNI 192.168.201.109@tcp101 [19859.424274] LNet: Removed LNI 192.168.201.109@tcp102 [19860.447171] LNet: Removed LNI 192.168.201.109@tcp103 [19860.452187] LNetError: 609724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19860.467896] LNetError: 609724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [19860.478551] LNet: Removed LNI 192.168.201.109@tcp104 [19861.535430] LNet: Removed LNI 192.168.201.109@tcp105 [19862.561042] LNet: Removed LNI 192.168.201.109@tcp106 [19863.583177] LNet: Removed LNI 192.168.201.109@tcp107 [19864.609571] LNet: Removed LNI 192.168.201.109@tcp108 [19864.614264] LNetError: 609724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19864.628452] LNetError: 609724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [19864.640275] LNet: Removed LNI 192.168.201.109@tcp109 [19865.697490] LNet: Removed LNI 192.168.201.109@tcp110 [19866.719156] LNet: Removed LNI 192.168.201.109@tcp111 [19867.743528] LNet: Removed LNI 192.168.201.109@tcp112 [19868.767444] LNet: Removed LNI 192.168.201.109@tcp113 [19869.791269] LNet: Removed LNI 192.168.201.109@tcp114 [19870.818131] LNet: Removed LNI 192.168.201.109@tcp115 [19871.839183] LNet: Removed LNI 192.168.201.109@tcp116 [19872.863584] LNet: Removed LNI 192.168.201.109@tcp117 [19872.867210] LNetError: 609724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19872.880232] LNetError: 609724:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [19872.889714] LNet: Removed LNI 192.168.201.109@tcp118 [19873.951321] LNet: Removed LNI 192.168.201.109@tcp119 [19874.975514] LNet: Removed LNI 192.168.201.109@tcp120 [19876.003194] LNet: Removed LNI 192.168.201.109@tcp121 [19876.005778] LNet: Removed LNI 192.168.201.109@tcp122 [19877.023590] LNet: Removed LNI 192.168.201.109@tcp123 [19878.047161] LNet: Removed LNI 192.168.201.109@tcp124 [19878.053137] LNet: Removed LNI 192.168.201.109@tcp125 [19878.057783] LNet: Removed LNI 192.168.201.109@tcp126 [19878.065446] LNet: Removed LNI 192.168.201.109@tcp127 [19878.076570] LNet: Removed LNI 192.168.201.109@tcp128 [19878.082906] LNet: Removed LNI 192.168.201.109@tcp129 [19878.088691] LNet: Removed LNI 192.168.201.109@tcp130 [19878.101482] LNet: Removed LNI 192.168.201.109@tcp131 [19878.103311] LNet: Removed LNI 192.168.201.109@tcp132 [19878.109215] LNet: Removed LNI 192.168.201.109@tcp133 [19878.116286] LNet: Removed LNI 192.168.201.109@tcp134 [19878.129691] LNet: Removed LNI 192.168.201.109@tcp135 [19878.942225] Key type .llcrypt unregistered [19878.947057] Key type ._llcrypt unregistered [19910.827281] Key type ._llcrypt registered [19910.829841] Key type .llcrypt registered [19910.982923] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [19931.689194] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [19933.434805] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19933.527693] alg: No test for adler32 (adler32-zlib) [19934.691961] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [19935.104232] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [19936.977232] Key type lgssc registered [19938.968714] Lustre: Echo OBD driver; http://www.lustre.org/ [19985.361417] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 06:22:28 (1788171748) [20008.356841] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [20011.685720] Key type lgssc unregistered [20012.026891] LNet: 613818:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20012.034958] LNetError: 613818:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20012.052297] LNet: Removed LNI 192.168.201.109@tcp [20013.179964] Key type .llcrypt unregistered [20013.182716] Key type ._llcrypt unregistered [20033.921801] Key type ._llcrypt registered [20033.924573] Key type .llcrypt registered [20034.134636] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [20036.326936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20036.414174] alg: No test for adler32 (adler32-zlib) [20037.430465] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [20037.654073] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [20039.311199] Key type lgssc registered [20041.217445] Lustre: Echo OBD driver; http://www.lustre.org/ [20056.045212] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20056.068191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20057.340774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20057.384366] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20057.477497] Lustre: lustre-MDT0000: new disk, initializing [20057.562740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20057.581909] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20063.054042] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [20077.175897] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20077.433936] Lustre: 615036:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20077.550570] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20077.555020] Lustre: Skipped 1 previous similar message [20077.686355] Lustre: lustre-MDT0001: new disk, initializing [20077.844309] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20077.930249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20077.941064] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20084.087537] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [20095.329627] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20102.315159] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20110.217044] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20110.608302] Lustre: lustre-OST0000: new disk, initializing [20110.615696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20110.626795] Lustre: 617001:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20110.749848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20117.038511] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [20117.547675] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20117.572562] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20117.678700] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20128.190399] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20132.275703] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing lnet_if_list [20138.304944] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [20138.356978] LNet: Added LNI 192.168.201.109@tcp10 [8/256/0/180] [20148.196833] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [20148.247209] LNet: Added LNI 192.168.201.109@tcp11 [8/256/0/180] [20154.097994] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [20154.131786] LNet: Added LNI 192.168.201.109@tcp12 [8/256/0/180] [20158.851931] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [20158.877549] LNet: Added LNI 192.168.201.109@tcp13 [8/256/0/180] [20163.936126] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [20163.983991] LNet: Added LNI 192.168.201.109@tcp14 [8/256/0/180] [20168.879970] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [20168.932155] LNet: Added LNI 192.168.201.109@tcp15 [8/256/0/180] [20173.716871] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [20173.744521] LNet: Added LNI 192.168.201.109@tcp16 [8/256/0/180] [20180.104705] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [20180.153322] LNet: Added LNI 192.168.201.109@tcp17 [8/256/0/180] [20185.982693] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [20186.034632] LNet: Added LNI 192.168.201.109@tcp18 [8/256/0/180] [20192.515234] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [20192.570880] LNet: Added LNI 192.168.201.109@tcp19 [8/256/0/180] [20198.873514] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [20198.905465] LNet: Added LNI 192.168.201.109@tcp20 [8/256/0/180] [20206.239568] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [20206.311257] LNet: Added LNI 192.168.201.109@tcp21 [8/256/0/180] [20213.132492] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [20213.185486] LNet: Added LNI 192.168.201.109@tcp22 [8/256/0/180] [20219.308404] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [20219.331938] LNet: Added LNI 192.168.201.109@tcp23 [8/256/0/180] [20226.338722] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [20226.365628] LNet: Added LNI 192.168.201.109@tcp24 [8/256/0/180] [20233.115440] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [20233.170787] LNet: Added LNI 192.168.201.109@tcp25 [8/256/0/180] [20238.910093] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [20238.971077] LNet: Added LNI 192.168.201.109@tcp26 [8/256/0/180] [20244.877393] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [20244.907278] LNet: Added LNI 192.168.201.109@tcp27 [8/256/0/180] [20251.594304] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [20251.637519] LNet: Added LNI 192.168.201.109@tcp28 [8/256/0/180] [20258.873879] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [20258.908151] LNet: Added LNI 192.168.201.109@tcp29 [8/256/0/180] [20267.050715] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [20267.122690] LNet: Added LNI 192.168.201.109@tcp30 [8/256/0/180] [20274.207616] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [20274.254882] LNet: Added LNI 192.168.201.109@tcp31 [8/256/0/180] [20281.335471] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [20281.359425] LNet: Added LNI 192.168.201.109@tcp32 [8/256/0/180] [20289.235695] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [20289.273528] LNet: Added LNI 192.168.201.109@tcp33 [8/256/0/180] [20298.114992] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [20298.151555] LNet: Added LNI 192.168.201.109@tcp34 [8/256/0/180] [20306.682319] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [20306.700966] LNet: Added LNI 192.168.201.109@tcp35 [8/256/0/180] [20314.823108] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [20314.878234] LNet: Added LNI 192.168.201.109@tcp36 [8/256/0/180] [20323.735080] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [20323.786630] LNet: Added LNI 192.168.201.109@tcp37 [8/256/0/180] [20333.553575] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [20333.583284] LNet: Added LNI 192.168.201.109@tcp38 [8/256/0/180] [20342.866349] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [20342.932646] LNet: Added LNI 192.168.201.109@tcp39 [8/256/0/180] [20355.360628] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [20355.412634] LNet: Added LNI 192.168.201.109@tcp40 [8/256/0/180] [20364.765948] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [20364.787046] LNet: Added LNI 192.168.201.109@tcp41 [8/256/0/180] [20373.843807] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [20373.876635] LNet: Added LNI 192.168.201.109@tcp42 [8/256/0/180] [20384.132988] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [20384.160570] LNet: Added LNI 192.168.201.109@tcp43 [8/256/0/180] [20395.038181] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [20395.117827] LNet: Added LNI 192.168.201.109@tcp44 [8/256/0/180] [20405.227666] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [20405.259205] LNet: Added LNI 192.168.201.109@tcp45 [8/256/0/180] [20416.465543] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [20416.523489] LNet: Added LNI 192.168.201.109@tcp46 [8/256/0/180] [20426.693242] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [20426.726859] LNet: Added LNI 192.168.201.109@tcp47 [8/256/0/180] [20436.555920] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [20436.582273] LNet: Added LNI 192.168.201.109@tcp48 [8/256/0/180] [20447.979950] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [20448.004868] LNet: Added LNI 192.168.201.109@tcp49 [8/256/0/180] [20458.771722] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [20458.805047] LNet: Added LNI 192.168.201.109@tcp50 [8/256/0/180] [20481.507629] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20481.513495] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20481.525595] Lustre: Skipped 2 previous similar messages [20481.531197] Lustre: Skipped 1 previous similar message [20484.421482] Lustre: server umount lustre-MDT0000 complete [20488.936821] LustreError: 615027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172254 with bad export cookie 8002666880165100225 [20488.940493] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20488.960244] LustreError: 615027:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [20489.520720] Lustre: server umount lustre-MDT0001 complete [20508.127450] Lustre: 614304:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172257/real 1788172257] req@000000000fb16c13 x1875034036696576/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [20508.163500] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20510.176720] Lustre: 614305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172259/real 1788172259] req@000000009ab3635c x1875034036696704/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172275 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [20513.247308] Lustre: 614306:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172262/real 1788172262] req@000000007ea6d8d8 x1875034036696832/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [20515.131704] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [20515.296231] Lustre: 614307:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172264/real 1788172264] req@000000004b96bb1a x1875034036697088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172280 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [20519.393747] Lustre: 614305:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788172269/real 1788172269] req@00000000fc69d8c5 x1875034036697216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788172285 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [20527.433804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20534.450793] LustreError: 616994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20534.489536] LustreError: 616994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [20534.569988] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20536.609850] LustreError: 616995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20536.632542] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [20537.634032] LustreError: 616994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20542.375792] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [20542.951367] LustreError: 616995:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20548.081119] LustreError: 616994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20553.677758] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20553.967082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20554.990932] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [20558.980850] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [20570.430827] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20577.608922] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20610.534505] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20610.544826] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20610.579722] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20610.594601] Lustre: Skipped 1 previous similar message [20611.559547] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [20613.400510] Lustre: server umount lustre-OST0000 complete [20620.768283] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20620.774450] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20620.784581] Lustre: Skipped 1 previous similar message [20620.796191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20620.803556] Lustre: Skipped 2 previous similar messages [20624.189481] Lustre: server umount lustre-MDT0000 complete [20626.913622] LustreError: 641259:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20626.954275] LustreError: 641259:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [20628.679635] LustreError: 618213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172394 with bad export cookie 8002666880165102927 [20628.693365] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20628.700475] LustreError: 618213:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [20629.019112] Lustre: server umount lustre-MDT0001 complete [20641.594278] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [20644.516841] Key type lgssc unregistered [20644.973737] LNet: 644059:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20644.979641] LNetError: 644059:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20644.987980] LNet: Removed LNI 192.168.201.109@tcp [20644.991160] LNet: Removed LNI 192.168.201.109@tcp10 [20644.994482] LNet: Removed LNI 192.168.201.109@tcp11 [20644.999691] LNet: Removed LNI 192.168.201.109@tcp12 [20645.003380] LNet: Removed LNI 192.168.201.109@tcp13 [20645.010862] LNet: Removed LNI 192.168.201.109@tcp14 [20645.014356] LNet: Removed LNI 192.168.201.109@tcp15 [20645.016172] LNet: Removed LNI 192.168.201.109@tcp16 [20645.021195] LNet: Removed LNI 192.168.201.109@tcp17 [20645.025191] LNet: Removed LNI 192.168.201.109@tcp18 [20645.031525] LNet: Removed LNI 192.168.201.109@tcp19 [20645.036991] LNet: Removed LNI 192.168.201.109@tcp20 [20645.040763] LNet: Removed LNI 192.168.201.109@tcp21 [20645.043436] LNet: Removed LNI 192.168.201.109@tcp22 [20645.046268] LNet: Removed LNI 192.168.201.109@tcp23 [20645.054430] LNet: Removed LNI 192.168.201.109@tcp24 [20645.058157] LNet: Removed LNI 192.168.201.109@tcp25 [20645.068957] LNet: Removed LNI 192.168.201.109@tcp26 [20645.079243] LNet: Removed LNI 192.168.201.109@tcp27 [20645.083180] LNet: Removed LNI 192.168.201.109@tcp28 [20645.096151] LNet: Removed LNI 192.168.201.109@tcp29 [20645.098223] LNet: Removed LNI 192.168.201.109@tcp30 [20645.101764] LNet: Removed LNI 192.168.201.109@tcp31 [20645.104315] LNet: Removed LNI 192.168.201.109@tcp32 [20645.106912] LNet: Removed LNI 192.168.201.109@tcp33 [20645.117589] LNet: Removed LNI 192.168.201.109@tcp34 [20645.127353] LNet: Removed LNI 192.168.201.109@tcp35 [20645.134273] LNet: Removed LNI 192.168.201.109@tcp36 [20645.138205] LNet: Removed LNI 192.168.201.109@tcp37 [20645.141298] LNet: Removed LNI 192.168.201.109@tcp38 [20645.144342] LNet: Removed LNI 192.168.201.109@tcp39 [20645.155018] LNet: Removed LNI 192.168.201.109@tcp40 [20645.158172] LNet: Removed LNI 192.168.201.109@tcp41 [20645.161104] LNet: Removed LNI 192.168.201.109@tcp42 [20645.164210] LNet: Removed LNI 192.168.201.109@tcp43 [20645.169878] LNet: Removed LNI 192.168.201.109@tcp44 [20645.175972] LNet: Removed LNI 192.168.201.109@tcp45 [20645.178901] LNet: Removed LNI 192.168.201.109@tcp46 [20645.185198] LNet: Removed LNI 192.168.201.109@tcp47 [20645.188187] LNet: Removed LNI 192.168.201.109@tcp48 [20645.189854] LNet: Removed LNI 192.168.201.109@tcp49 [20645.203399] LNet: Removed LNI 192.168.201.109@tcp50 [20646.102734] Key type .llcrypt unregistered [20646.104870] Key type ._llcrypt unregistered [20667.531683] Key type ._llcrypt registered [20667.535358] Key type .llcrypt registered [20667.774163] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [20686.972848] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [20688.366173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20688.408110] alg: No test for adler32 (adler32-zlib) [20689.674276] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [20689.976278] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [20691.689059] Key type lgssc registered [20693.153590] Lustre: Echo OBD driver; http://www.lustre.org/ [20732.648241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20732.669684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20733.986715] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20734.036061] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20734.163217] Lustre: lustre-MDT0000: new disk, initializing [20734.260164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20734.280673] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20738.558279] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [20749.594471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20749.720230] Lustre: 647258:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20749.771182] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20749.780376] Lustre: Skipped 1 previous similar message [20749.848655] Lustre: lustre-MDT0001: new disk, initializing [20749.977569] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20750.023424] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20750.035424] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20754.467847] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [20765.445559] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20772.037703] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20780.734033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20780.952165] Lustre: *** cfs_fail_loc=90f, val=0*** [20781.163204] Lustre: lustre-OST0000: new disk, initializing [20781.173447] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20781.183579] Lustre: 649277:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20781.345376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20786.760681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20786.770716] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20786.881913] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20789.182571] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [20799.214159] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20812.769893] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [20812.784597] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20812.786171] LustreError: Skipped 1 previous similar message [20812.792356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20812.810147] Lustre: Skipped 1 previous similar message [20815.655083] Lustre: server umount lustre-OST0000 complete [20821.472266] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20821.484892] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20821.501197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20821.503360] Lustre: Skipped 1 previous similar message [20823.017269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20823.018157] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20826.326834] Lustre: server umount lustre-MDT0000 complete [20830.187600] LustreError: 647251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172595 with bad export cookie 2243288349564011029 [20830.188691] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20830.200463] LustreError: 647251:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [20830.516270] Lustre: server umount lustre-MDT0001 complete [20842.873456] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [20846.174337] Key type lgssc unregistered [20846.543637] LNet: 651182:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20846.548916] LNetError: 651182:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20847.590741] LNet: Removed LNI 192.168.201.109@tcp [20848.284672] Key type .llcrypt unregistered [20848.287891] Key type ._llcrypt unregistered [20871.939591] Key type ._llcrypt registered [20871.943887] Key type .llcrypt registered [20872.101743] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [20873.338714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20873.419779] alg: No test for adler32 (adler32-zlib) [20874.548917] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [20874.765078] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [20876.407325] Key type lgssc registered [20877.706110] Lustre: Echo OBD driver; http://www.lustre.org/ [20886.581620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20886.597437] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20888.108768] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20892.680801] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [20898.285340] LustreError: 652455:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20898.312294] LustreError: 652455:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [20902.097898] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20902.523388] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20907.408364] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [20916.898361] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20924.895589] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20933.553042] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20933.804481] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20941.261639] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [20949.157423] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20971.348191] loop0: detected capacity change from 0 to 204800000 [20971.617552] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20971.636980] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20971.659736] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20977.990334] loop0: detected capacity change from 0 to 204800000 [20978.270193] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20978.283028] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20978.303865] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20983.780218] loop0: detected capacity change from 0 to 204800000 [20988.884757] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20988.997164] Lustre: fs15246-OST0000: new disk, initializing [20989.000695] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [20989.004432] Lustre: 656070:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20989.041268] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20995.426226] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [21005.578764] loop1: detected capacity change from 0 to 204800000 [21011.733602] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [21011.918055] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [21012.045883] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [21012.212577] Lustre: fs15246-MDT0000: new disk, initializing [21012.451540] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [21012.489162] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [21012.631461] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [21013.664132] Lustre: 657191:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000012c29ceb x1875034914209792/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788172795 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [21017.045556] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [21022.765574] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [21022.780560] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [21022.821666] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [21041.160906] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [21041.179160] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [21046.027240] Lustre: server umount fs15246-MDT0000 complete [21046.246761] LustreError: 652449:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [21051.372629] LustreError: 652451:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [21054.219287] Lustre: server umount fs15246-OST0000 complete [21071.332782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [21071.337509] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [21071.343430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [21071.356592] Lustre: Skipped 1 previous similar message [21072.661483] Lustre: server umount lustre-OST0000 complete [21076.963192] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [21076.985879] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [21076.988679] Lustre: Skipped 1 previous similar message [21081.572140] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [21081.575828] LustreError: Skipped 1 previous similar message [21081.585804] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [21081.599784] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [21082.939104] Lustre: server umount lustre-MDT0000 complete [21086.751852] LustreError: 652433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788172852 with bad export cookie 9007587458257399513 [21086.780595] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [21087.077234] Lustre: server umount lustre-MDT0001 complete [21098.096075] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [21100.536867] Key type lgssc unregistered [21100.771883] LNet: 660289:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [21100.787896] LNetError: 660289:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [21100.822742] LNet: Removed LNI 192.168.201.109@tcp [21101.605987] Key type .llcrypt unregistered [21101.607989] Key type ._llcrypt unregistered [21120.850068] Key type ._llcrypt registered [21120.852468] Key type .llcrypt registered [21120.943235] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [21122.350927] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [21122.387402] alg: No test for adler32 (adler32-zlib) [21123.550798] Lustre: Lustre: Build Version: 2.17.55_27_gdd19d92 [21123.765898] LNet: Added LNI 192.168.201.109@tcp [8/256/0/180] [21125.409518] Key type lgssc registered [21126.460693] Lustre: Echo OBD driver; http://www.lustre.org/ [21165.419057] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [21176.716890] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [21213.133975] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [21213.198240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [21214.528338] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [21214.553814] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [21214.668796] Lustre: lustre-MDT0000: new disk, initializing [21214.769280] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [21214.787557] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [21219.526374] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [21230.127102] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [21230.348545] Lustre: 664757:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [21230.403231] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [21230.407205] Lustre: Skipped 1 previous similar message [21230.500725] Lustre: lustre-MDT0001: new disk, initializing [21230.623503] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [21230.651877] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [21230.661722] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [21234.897048] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [21245.301851] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [21250.387387] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [21257.288343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [21257.659755] Lustre: lustre-OST0000: new disk, initializing [21257.663537] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [21257.671682] Lustre: 666726:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [21257.757476] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [21264.172618] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_default_debug -1 all [21267.061218] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [21267.069829] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [21267.248139] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [21274.282995] Lustre: DEBUG MARKER: oleg109-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [21279.237967] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [21279.591545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [21285.295090] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [21285.577732] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [21293.026326] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [21293.047244] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [21293.048200] LustreError: Skipped 1 previous similar message [21293.052435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [21293.110218] Lustre: Skipped 1 previous similar message [21294.906830] Lustre: server umount lustre-OST0000 complete [21302.240875] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [21302.257345] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [21302.279472] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [21302.292473] Lustre: Skipped 1 previous similar message [21303.264549] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [21306.294870] Lustre: server umount lustre-MDT0000 complete [21309.782330] LustreError: 664750:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788173075 with bad export cookie 5266882937428156537 [21309.797873] LustreError: MGC192.168.201.109@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [21310.181825] Lustre: server umount lustre-MDT0001 complete [21342.131895] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing set_hostid [21356.970269] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing load_modules_local [21389.467976] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 21098 sec ========= 06:45:52 (1788173152) [21391.586933] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 06:45:54 (1788173154) === [21395.188718] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 06:45:58 (1788173158) === [21437.438404] Lustre: DEBUG MARKER: oleg109-server.virtnet: executing unload_modules_local [21440.284718] Key type lgssc unregistered [21440.683751] LNet: 673395:0:(lib-ptl.c:977:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [21440.693329] LNetError: 673395:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [21440.713202] LNet: Removed LNI 192.168.201.109@tcp [21441.607720] Key type .llcrypt unregistered [21441.609413] Key type ._llcrypt unregistered