[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 469130621 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001009] APIC: Switch to symmetric I/O mode setup [ 0.003174] x2apic enabled [ 0.004005] Switched APIC routing to physical x2apic. [ 0.005013] kvm-guest: setup PV IPIs [ 0.008467] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010010] pid_max: default: 32768 minimum: 301 [ 0.011117] LSM: Security Framework initializing [ 0.012045] Yama: becoming mindful. [ 0.014035] SELinux: Initializing. [ 0.015052] *** VALIDATE selinux *** [ 0.023090] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.028094] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.029123] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030086] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031095] *** VALIDATE tmpfs *** [ 0.033437] *** VALIDATE proc *** [ 0.034225] *** VALIDATE cgroup *** [ 0.035009] *** VALIDATE cgroup2 *** [ 0.036247] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037144] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038008] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039025] Spectre V2 : User space: Vulnerable [ 0.040008] Speculative Store Bypass: Vulnerable [ 0.042845] debug: unmapping init [mem 0xffffffffba259000-0xffffffffba260fff] [ 0.044157] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.045672] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.046020] ... version: 2 [ 0.047011] ... bit width: 48 [ 0.048010] ... generic registers: 4 [ 0.049010] ... value mask: 0000ffffffffffff [ 0.050011] ... max period: 00007fffffffffff [ 0.051012] ... fixed-purpose events: 3 [ 0.052011] ... event mask: 000000070000000f [ 0.053272] rcu: Hierarchical SRCU implementation. [ 0.055361] smp: Bringing up secondary CPUs ... [ 0.056512] x86: Booting SMP configuration: [ 0.057018] .... node #0, CPUs: #1 #2 #3 [ 0.060459] smp: Brought up 1 node, 4 CPUs [ 0.062010] smpboot: Max logical packages: 1 [ 0.063015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.148016] node 0 deferred pages initialised in 82ms [ 0.152093] devtmpfs: initialized [ 0.154259] x86/mm: Memory block size: 128MB [ 0.157861] gcov: version magic: 0x41383552 [ 0.159362] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.163069] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.165296] pinctrl core: initialized pinctrl subsystem [ 0.167152] [ 0.167794] ************************************************************* [ 0.171014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.173009] ** ** [ 0.175010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.176010] ** ** [ 0.178010] ** This means that this kernel is built to expose internal ** [ 0.181012] ** IOMMU data structures, which may compromise security on ** [ 0.184011] ** your system. ** [ 0.186013] ** ** [ 0.189014] ** If you see this message and you are not debugging the ** [ 0.191011] ** kernel, report this immediately to your vendor! ** [ 0.194013] ** ** [ 0.197013] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.199014] ************************************************************* [ 0.202589] NET: Registered protocol family 16 [ 0.204467] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.207055] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.210056] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.215016] cpuidle: using governor menu [ 0.217536] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.219388] PCI: Using configuration type 1 for base access [ 0.222108] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.232051] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.234069] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.240030] cryptd: max_cpu_qlen set to 1000 [ 0.245307] ACPI: Added _OSI(Module Device) [ 0.246012] ACPI: Added _OSI(Processor Device) [ 0.247009] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.248009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.252438] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.258529] ACPI: Interpreter enabled [ 0.259044] ACPI: PM: (supports S0 S3 S4 S5) [ 0.260009] ACPI: Using IOAPIC for interrupt routing [ 0.262088] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.265379] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.274349] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.276028] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.278012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.280062] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.284441] acpiphp: Slot [2] registered [ 0.286077] acpiphp: Slot [5] registered [ 0.287106] acpiphp: Slot [6] registered [ 0.288076] acpiphp: Slot [7] registered [ 0.290072] acpiphp: Slot [8] registered [ 0.291080] acpiphp: Slot [9] registered [ 0.292067] acpiphp: Slot [10] registered [ 0.293071] acpiphp: Slot [3] registered [ 0.294141] acpiphp: Slot [4] registered [ 0.295069] acpiphp: Slot [11] registered [ 0.297065] acpiphp: Slot [12] registered [ 0.299094] acpiphp: Slot [13] registered [ 0.300090] acpiphp: Slot [14] registered [ 0.302110] acpiphp: Slot [15] registered [ 0.304086] acpiphp: Slot [16] registered [ 0.305084] acpiphp: Slot [17] registered [ 0.307086] acpiphp: Slot [18] registered [ 0.309083] acpiphp: Slot [19] registered [ 0.311095] acpiphp: Slot [20] registered [ 0.313082] acpiphp: Slot [21] registered [ 0.315089] acpiphp: Slot [22] registered [ 0.316081] acpiphp: Slot [23] registered [ 0.317065] acpiphp: Slot [24] registered [ 0.319067] acpiphp: Slot [25] registered [ 0.320079] acpiphp: Slot [26] registered [ 0.321065] acpiphp: Slot [27] registered [ 0.323067] acpiphp: Slot [28] registered [ 0.324073] acpiphp: Slot [29] registered [ 0.325061] acpiphp: Slot [30] registered [ 0.326079] acpiphp: Slot [31] registered [ 0.328054] PCI host bridge to bus 0000:00 [ 0.329011] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.332015] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.334014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.337017] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.340011] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.343026] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.345190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.349084] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.353147] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.362013] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.367049] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.371022] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.373012] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.376013] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.379483] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.381891] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.385038] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.388861] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.394013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.409000] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.414014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.421236] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.446014] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.469014] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.541015] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.568000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.578014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.589000] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.623014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.643453] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.655013] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.670020] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.724014] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.747757] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.766011] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.787012] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.812011] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.825000] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.844014] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.855012] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.883014] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.891545] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.901000] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.910014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.933011] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.942901] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.944362] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.947316] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.949283] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.951158] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.958057] iommu: Default domain type: Passthrough [ 0.960384] SCSI subsystem initialized [ 0.962157] ACPI: bus type USB registered [ 0.964097] usbcore: registered new interface driver usbfs [ 0.967065] usbcore: registered new interface driver hub [ 0.969059] usbcore: registered new device driver usb [ 0.971168] pps_core: LinuxPPS API ver. 1 registered [ 0.973009] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.978058] PTP clock support registered [ 0.980175] EDAC MC: Ver: 3.0.0 [ 0.981782] PCI: Using ACPI for IRQ routing [ 0.982752] NetLabel: Initializing [ 0.983010] NetLabel: domain hash size = 128 [ 0.984008] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.985088] NetLabel: unlabeled traffic allowed by default [ 0.987103] vgaarb: loaded [ 0.988247] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.989011] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.994204] clocksource: Switched to clocksource kvm-clock [ 1.107380] VFS: Disk quotas dquot_6.6.0 [ 1.109272] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 1.112272] *** VALIDATE ramfs *** [ 1.113716] *** VALIDATE hugetlbfs *** [ 1.116257] pnp: PnP ACPI init [ 1.118858] pnp: PnP ACPI: found 6 devices [ 1.146217] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 1.149021] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 1.150787] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 1.152688] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 1.154762] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 1.157512] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 1.160831] NET: Registered protocol family 2 [ 1.163708] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 1.169327] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 1.173167] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 1.178627] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 1.182281] TCP: Hash tables configured (established 65536 bind 65536) [ 1.185392] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.188861] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.192029] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.195407] NET: Registered protocol family 1 [ 1.198958] RPC: Registered named UNIX socket transport module. [ 1.201240] RPC: Registered udp transport module. [ 1.203073] RPC: Registered tcp transport module. [ 1.204720] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.207198] NET: Registered protocol family 44 [ 1.208676] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.210734] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.212722] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.214968] PCI: CLS 0 bytes, default 64 [ 1.216507] Unpacking initramfs... [ 2.634529] debug: unmapping init [mem 0xffff9e757cc54000-0xffff9e757ffbffff] [ 2.640683] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.642894] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.646657] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 3.131548] Initialise system trusted keyrings [ 3.133458] Key type blacklist registered [ 3.137957] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.148255] zbud: loaded [ 3.151890] *** VALIDATE nfs *** [ 3.153674] *** VALIDATE nfs4 *** [ 3.156036] pstore: using deflate compression [ 3.161280] Platform Keyring initialized [ 3.295866] NET: Registered protocol family 38 [ 3.297701] Key type asymmetric registered [ 3.299142] Asymmetric key parser 'x509' registered [ 3.300814] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.303595] io scheduler mq-deadline registered [ 3.305263] io scheduler kyber registered [ 3.306777] io scheduler bfq registered [ 3.308669] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.311659] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.314231] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.316853] ACPI: Power Button [PWRF] [ 3.322515] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.329732] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.366166] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.377249] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.427141] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.455028] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.482974] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.486723] Non-volatile memory driver v1.3 [ 3.487831] Linux agpgart interface v0.103 [ 3.521431] virtio_blk virtio1: [vda] 145872 512-byte logical blocks (74.7 MB/71.2 MiB) [ 3.524920] vda: detected capacity change from 0 to 74686464 [ 3.550746] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.553645] vdb: detected capacity change from 0 to 1073741824 [ 3.578721] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.581758] vdc: detected capacity change from 0 to 2621440000 [ 3.601603] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.605291] vdd: detected capacity change from 0 to 2621440000 [ 3.629294] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.633299] vde: detected capacity change from 0 to 4294967296 [ 3.655192] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.659665] vdf: detected capacity change from 0 to 4294967296 [ 3.672211] libphy: Fixed MDIO Bus: probed [ 3.678248] usbcore: registered new interface driver usbserial_generic [ 3.681363] usbserial: USB Serial support registered for generic [ 3.684612] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.690092] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.692337] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.695544] mousedev: PS/2 mouse device common for all mice [ 3.698991] rtc_cmos 00:05: RTC can wake from S4 [ 3.703681] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.704555] rtc_cmos 00:05: registered as rtc0 [ 3.708940] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.712504] intel_pstate: CPU model not supported [ 3.714982] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.719165] hid: raw HID events driver (C) Jiri Kosina [ 3.721984] usbcore: registered new interface driver usbhid [ 3.723934] usbhid: USB HID core driver [ 3.725590] drop_monitor: Initializing network drop monitor service [ 3.725645] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.727232] Initializing XFRM netlink socket [ 3.732272] NET: Registered protocol family 10 [ 3.734837] Segment Routing with IPv6 [ 3.736350] NET: Registered protocol family 17 [ 3.738649] mpls_gso: MPLS GSO support [ 3.745400] RAS: Correctable Errors collector initialized. [ 3.747535] AVX version of gcm_enc/dec engaged. [ 3.749250] AES CTR mode by8 optimization enabled [ 3.825877] sched_clock: Marking stable (3825854856, 0)->(4717653296, -891798440) [ 3.829429] registered taskstats version 1 [ 3.832182] Loading compiled-in X.509 certificates [ 3.833969] zswap: loaded using pool lzo/zbud [ 3.858111] Key type big_key registered [ 3.870943] Key type encrypted registered [ 3.872414] ima: No TPM chip found, activating TPM-bypass! [ 3.874868] ima: Allocated hash algorithm: sha1 [ 3.876701] ima: No architecture policies found [ 3.878574] evm: Initialising EVM extended attributes: [ 3.880628] evm: security.selinux [ 3.881861] evm: security.ima [ 3.882922] evm: security.capability [ 3.884265] evm: HMAC attrs: 0x1 [ 3.887215] rtc_cmos 00:05: setting system clock to 2026-07-30 22:31:20 UTC (1785450680) [ 3.893764] debug: unmapping init [mem 0xffffffffbb203000-0xffffffffbb3fffff] [ 3.896818] debug: unmapping init [mem 0xffffffffb9f82000-0xffffffffba258fff] [ 3.905396] Write protecting the kernel read-only data: 28672k [ 3.909619] debug: unmapping init [mem 0xffffffffb8603000-0xffffffffb87fffff] [ 3.912983] debug: unmapping init [mem 0xffffffffb8f14000-0xffffffffb8ffffff] [ 3.948590] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.958206] systemd[1]: Detected virtualization kvm. [ 3.960023] systemd[1]: Detected architecture x86-64. [ 3.962099] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.989690] systemd[1]: No hostname configured. [ 3.991618] systemd[1]: Set hostname to . [ 3.993694] random: systemd: uninitialized urandom read (16 bytes read) [ 3.996356] systemd[1]: Initializing machine ID from random generator. [ 4.039420] random: ln: uninitialized urandom read (6 bytes read) [ 4.117748] random: systemd: uninitialized urandom read (16 bytes read) [ 4.121092] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 4.126045] systemd[1]: Reached target Timers. [ OK ] Reached target Timers. [ 4.130050] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Local File Systems. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting Setup Virtual Console... [ OK ] Reached target Slices. [ OK ] Reached target Swap. Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create Volatile Files and Directories... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.726410] device-mapper: uevent: version 1.0.3 [ 4.728772] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.482611] random: fast init done [ 5.540209] virtio_net virtio0 ens2: renamed from eth0 [ 5.704285] scsi host0: ata_piix [ 5.711463] scsi host1: ata_piix [ 5.712967] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.715224] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.274303] dracut-initqueue[591]: RTNETLINK answers: File exists [ 10.395235] random: crng init done [ 10.396540] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.795429] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Paths. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.934934] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.173828] SELinux: Disabled at runtime. [ 12.231870] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.241249] systemd[1]: Detected virtualization kvm. [ 12.242890] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.803066] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.806575] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.811382] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.815751] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.819545] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.829701] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.838944] systemd[1]: Mounting Huge Pages File System... Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... Mounting Kernel Debug File System... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-serial\x2dgetty.slice. Starting Remount Root and Kernel File Systems... [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on[ 12.919854] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS udev Control Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Listening on Process Core Dump Socket. Starting Apply Kernel Variables... [ OK ] Reached target rpc_pipefs.target. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting udev Coldplug all Devices... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Started Journal Service. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted POSIX Message Queue File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... [ OK ] Mounted /mnt. [ OK [[ 13.325867] squashfs: version 4.0 (2009/01/31) Phillip Lougher 0m] Started udev Coldplug all Devices. [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.622550] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.636779] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.771232] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.785130] EDAC sbridge: Ver: 1.1.2 [ 15.429537] Key type dns_resolver registered [ 15.749975] NFS: Registering the id_resolver key type [ 15.752343] Key type id_resolver registered [ 15.754137] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. Starting Network Manager... Starting Restore /run/initramfs on shutdown... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Notify NFS peers of a restart... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg329-server login: [ 84.763018] libcfs: loading out-of-tree module taints kernel. [ 84.816761] Key type ._llcrypt registered [ 84.820616] Key type .llcrypt registered [ 84.952849] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 110.355357] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 112.540150] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 112.572274] alg: No test for adler32 (adler32-zlib) [ 114.251700] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 115.796900] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 117.904224] Key type lgssc registered [ 119.604272] hrtimer: interrupt took 4290974 ns [ 120.645399] Lustre: Echo OBD driver; http://www.lustre.org/ [ 151.991975] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 210.994216] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 229.936494] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 229.982533] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 231.909239] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 231.967304] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 232.287914] Lustre: lustre-MDT0000: new disk, initializing [ 232.527045] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.606155] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 237.728343] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 257.047583] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 257.263534] Lustre: 6546:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 257.378953] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 257.396335] Lustre: Skipped 1 previous similar message [ 257.597480] Lustre: lustre-MDT0001: new disk, initializing [ 257.784791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 257.847640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 257.869129] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 265.622063] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 272.270588] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 286.499442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 286.932907] Lustre: lustre-OST0000: new disk, initializing [ 286.944721] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 286.961420] Lustre: 8486:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 287.106722] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 295.993980] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 296.022217] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 296.137440] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 296.192576] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 318.168036] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 318.406291] Lustre: lustre-OST0001: new disk, initializing [ 318.422278] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 318.434226] Lustre: 9559:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 318.569070] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 323.715899] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 323.727070] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 323.847886] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 328.697517] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 345.720333] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 356.488267] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 367.725855] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing check_logdir /tmp/testlogs/ [ 373.660826] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing yml_node [ 379.164579] Lustre: DEBUG MARKER: Client: 2.17.56.1 [ 382.832105] Lustre: DEBUG MARKER: MDS: 2.17.56.1 [ 386.175996] Lustre: DEBUG MARKER: OSS: 2.17.56.1 [ 389.492585] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jul 30 18:37:42 EDT 2026 [ 412.764803] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 414.466661] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 431.586453] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 431.588066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 431.612240] Lustre: Skipped 3 previous similar messages [ 431.619061] Lustre: Skipped 2 previous similar messages [ 436.710079] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 436.719324] Lustre: Skipped 1 previous similar message [ 438.011389] Lustre: server umount lustre-MDT0000 complete [ 443.645140] LustreError: 6540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785451120 with bad export cookie 576820158640738784 [ 443.650485] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 443.661291] LustreError: 6540:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 446.945671] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 446.946377] LustreError: 7825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 446.948451] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 446.968892] Lustre: Skipped 1 previous similar message [ 447.033373] LustreError: 7825:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 448.994218] LustreError: 6553:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 448.994691] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 449.017970] LustreError: 6553:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 449.052434] Lustre: Skipped 2 previous similar messages [ 450.475446] Lustre: server umount lustre-MDT0001 complete [ 457.362785] Lustre: server umount lustre-OST0000 complete [ 462.969203] Lustre: server umount lustre-OST0001 complete [ 472.369392] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 488.207506] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 537.691867] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 538.129620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 538.169846] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 538.299555] Lustre: lustre-MDT0000: new disk, initializing [ 538.450212] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 538.479757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 544.907898] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 558.095355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 558.315140] Lustre: 15140:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 558.442591] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 558.453758] Lustre: Skipped 1 previous similar message [ 558.659919] Lustre: lustre-MDT0001: new disk, initializing [ 558.872249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 558.928874] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 558.947887] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 566.353943] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 580.805145] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 588.858331] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 598.576737] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 599.082052] Lustre: lustre-OST0000: new disk, initializing [ 599.092824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 599.117567] Lustre: 17110:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 599.311283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 601.145460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 601.157897] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 601.265789] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 606.527226] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 617.440656] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 623.376656] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 623.619952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 630.811305] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 631.310207] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 637.410487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 637.423226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 637.428297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 637.453029] Lustre: Skipped 1 previous similar message [ 637.472652] Lustre: Skipped 1 previous similar message [ 641.287952] Lustre: server umount lustre-OST0000 complete [ 647.653271] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 647.659639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 647.669552] Lustre: Skipped 1 previous similar message [ 647.700617] Lustre: Skipped 1 previous similar message [ 654.135570] Lustre: server umount lustre-MDT0000 complete [ 657.898508] LustreError: 18373:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 657.923072] LustreError: 18373:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 659.125788] LustreError: 15133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785451335 with bad export cookie 576820158640740317 [ 659.129840] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 659.137786] LustreError: 15133:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 659.431714] Lustre: server umount lustre-MDT0001 complete [ 665.585487] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:42:19 (1785451339) [ 669.883494] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 672.327228] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:42:26 (1785451346) [ 684.025413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 684.785516] LustreError: 19526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 684.969325] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 690.151933] LustreError: 19527:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 692.224353] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 700.387856] LustreError: 19526:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 700.410227] LustreError: 19526:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 702.558786] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 710.131897] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 720.801042] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 729.801977] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 741.363673] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 751.124280] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 762.386748] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 774.113383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 774.126674] LustreError: Skipped 1 previous similar message [ 774.132720] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 774.148603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 774.159541] Lustre: Skipped 2 previous similar messages [ 776.175526] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 778.259332] Lustre: server umount lustre-OST0000 complete [ 784.882700] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 784.898175] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 784.918549] Lustre: Skipped 1 previous similar message [ 789.585939] Lustre: server umount lustre-MDT0000 complete [ 791.530021] LustreError: 20452:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 791.554162] LustreError: 20452:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 794.209725] LustreError: 19507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785451470 with bad export cookie 576820158640741199 [ 794.215221] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 794.221254] LustreError: 19507:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 794.804688] Lustre: server umount lustre-MDT0001 complete [ 810.079305] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 814.168303] Key type lgssc unregistered [ 814.726556] LNet: 23066:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 814.737229] LNetError: 23066:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 814.760216] LNet: Removed LNI 192.168.203.129@tcp [ 815.960139] Key type .llcrypt unregistered [ 815.963133] Key type ._llcrypt unregistered [ 845.455990] Key type ._llcrypt registered [ 845.466880] Key type .llcrypt registered [ 845.615914] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 847.340629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 847.602809] alg: No test for adler32 (adler32-zlib) [ 849.166258] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 849.657132] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 851.536603] Key type lgssc registered [ 853.217249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 867.759480] loop: module loaded [ 867.928692] loop0: detected capacity change from 0 to 204800000 [ 868.292866] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 868.313065] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 869.200978] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 877.965649] loop0: detected capacity change from 0 to 204800000 [ 878.225618] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 878.258915] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 878.275399] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 885.202420] loop0: detected capacity change from 0 to 204800000 [ 893.122039] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 893.158776] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 894.640658] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 894.710106] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 894.855211] Lustre: lustre-MDT0000: new disk, initializing [ 894.982621] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 895.007759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 900.654660] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 911.112170] loop1: detected capacity change from 0 to 204800000 [ 918.173187] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 918.781748] Lustre: lustre-OST07c6: new disk, initializing [ 918.786911] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 918.791413] Lustre: Skipped 1 previous similar message [ 918.797132] Lustre: 26260:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 918.948614] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 924.297187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 924.310305] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 924.374569] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 927.299554] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 932.946535] Lustre: 25245:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 932.965576] Lustre: 25245:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 932.979646] Lustre: 25245:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 932.997894] Lustre: 25245:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 933.017898] Lustre: 25245:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 933.030065] Lustre: 25245:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 955.820297] Lustre: 25245:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 955.842910] Lustre: 25245:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 955.861167] Lustre: 25245:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 955.883402] Lustre: 25245:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 955.907820] Lustre: 25245:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 955.927106] Lustre: 25245:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 962.255899] Lustre: 25245:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 962.285374] Lustre: 25245:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 962.307521] Lustre: 25245:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 962.321064] Lustre: 25245:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 962.328809] Lustre: 25245:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 962.335789] Lustre: 25245:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 962.343152] Lustre: 25245:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 962.351670] Lustre: 25245:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 962.362168] Lustre: 25245:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 962.366929] Lustre: 25245:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 962.373254] Lustre: 25245:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 962.382114] Lustre: 25245:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 965.686724] Lustre: 26907:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 965.712542] Lustre: 26907:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 965.726970] Lustre: 26907:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 965.749161] Lustre: 26907:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 965.765721] Lustre: 26907:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 965.779837] Lustre: 26907:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 965.792042] Lustre: 26907:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 965.803336] Lustre: 26907:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 965.818689] Lustre: 26907:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 965.829371] Lustre: 26907:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 965.837805] Lustre: 26907:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 965.851194] Lustre: 26907:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 976.387064] Lustre: 25247:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 976.410095] Lustre: 25247:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 976.435237] Lustre: 25247:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 976.444196] Lustre: 25247:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 976.451783] Lustre: 25247:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 976.470746] Lustre: 25247:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 976.489482] Lustre: 25247:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 976.503677] Lustre: 25247:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 976.509685] Lustre: 25247:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 976.516387] Lustre: 25247:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 976.522284] Lustre: 25247:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 976.532249] Lustre: 25247:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 990.274390] Lustre: 25245:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 990.313905] Lustre: 25245:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 990.328229] Lustre: 25245:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 990.350922] Lustre: 25245:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 990.375023] Lustre: 25245:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 990.388406] Lustre: 25245:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 990.519680] Lustre: 25245:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 990.531134] Lustre: 25245:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 990.537226] Lustre: 25245:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 990.542343] Lustre: 25245:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 990.548130] Lustre: 25245:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 990.553647] Lustre: 25245:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 1036.770662] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1036.790396] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1041.893282] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 1042.837149] Lustre: server umount lustre-OST07c6 complete [ 1056.237818] Lustre: server umount lustre-MDT0000 complete [ 1081.050078] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1086.661700] Key type lgssc unregistered [ 1087.296646] LNet: 28828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1087.303623] LNetError: 28828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.389723] LNet: Removed LNI 192.168.203.129@tcp [ 1090.043334] Key type .llcrypt unregistered [ 1090.056151] Key type ._llcrypt unregistered [ 1127.140938] Key type ._llcrypt registered [ 1127.145384] Key type .llcrypt registered [ 1127.342115] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1128.565125] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1128.796882] alg: No test for adler32 (adler32-zlib) [ 1129.847297] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 1130.085041] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1131.763933] Key type lgssc registered [ 1133.550962] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1144.142281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1144.191666] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1145.814403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1151.945487] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1156.068582] LustreError: 30105:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1156.085617] LustreError: 30105:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1161.189523] LustreError: 30104:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1162.820147] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1163.353391] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1168.689836] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1178.128840] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1185.985911] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1195.610742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1196.342168] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1203.822833] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1213.156815] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1215.801189] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1241.574120] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1241.582831] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1241.590631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1241.611566] Lustre: Skipped 1 previous similar message [ 1241.627275] Lustre: Skipped 1 previous similar message [ 1245.487187] Lustre: server umount lustre-OST0000 complete [ 1251.810974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1251.828600] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1251.833662] Lustre: Skipped 1 previous similar message [ 1256.943139] LustreError: 30832:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1257.357676] Lustre: server umount lustre-MDT0000 complete [ 1261.976511] LustreError: 30085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785451938 with bad export cookie 15814090744394439826 [ 1261.985029] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1261.992706] LustreError: 30085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1262.418476] Lustre: server umount lustre-MDT0001 complete [ 1278.228922] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1282.006766] Key type lgssc unregistered [ 1282.362297] LNet: 33866:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1282.375847] LNetError: 33866:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1282.396889] LNet: Removed LNI 192.168.203.129@tcp [ 1283.958323] Key type .llcrypt unregistered [ 1283.963191] Key type ._llcrypt unregistered [ 1311.250975] Key type ._llcrypt registered [ 1311.252996] Key type .llcrypt registered [ 1311.403616] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1312.897344] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1312.999248] alg: No test for adler32 (adler32-zlib) [ 1314.313349] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 1314.853097] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1316.656150] Key type lgssc registered [ 1318.729594] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1329.658528] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1329.701368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1331.449536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1337.598745] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1340.897874] LustreError: 35144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1340.918715] LustreError: 35144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1346.022256] LustreError: 35145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1348.180777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1348.581873] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1353.865727] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1364.656941] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1373.111718] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1381.732165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1382.300138] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1391.473845] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1400.756877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1403.995418] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1419.235311] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1419.253836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.278676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1421.293218] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1421.296180] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1421.300351] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1421.308718] Lustre: Skipped 1 previous similar message [ 1425.554793] Lustre: server umount lustre-OST0000 complete [ 1431.529678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1431.531847] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1431.571701] Lustre: Skipped 1 previous similar message [ 1431.594080] Lustre: Skipped 1 previous similar message [ 1436.643964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1436.651928] Lustre: Skipped 1 previous similar message [ 1437.552951] Lustre: server umount lustre-MDT0000 complete [ 1441.766829] LustreError: 35144:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.787909] LustreError: 35144:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1442.004186] LustreError: 35123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785452118 with bad export cookie 4692517496659309415 [ 1442.011017] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1442.023889] LustreError: 35123:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1442.860948] Lustre: server umount lustre-MDT0001 complete [ 1458.010763] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1462.448985] Key type lgssc unregistered [ 1462.990605] LNet: 38703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1462.999837] LNetError: 38703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1464.045685] LNet: Removed LNI 192.168.203.129@tcp [ 1465.456579] Key type .llcrypt unregistered [ 1465.458197] Key type ._llcrypt unregistered [ 1496.311453] Key type ._llcrypt registered [ 1496.313467] Key type .llcrypt registered [ 1496.488846] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1498.389408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1498.436436] alg: No test for adler32 (adler32-zlib) [ 1499.664575] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 1500.026386] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1501.880223] Key type lgssc registered [ 1503.588813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1512.927887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1512.968457] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1514.804290] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1522.325603] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1525.229221] LustreError: 40123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.262293] LustreError: 40123:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1530.347553] LustreError: 40122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1535.461440] LustreError: 40123:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1537.409882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1538.036298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1543.706816] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1555.323565] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1566.212922] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1578.927783] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1579.777288] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1584.918718] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1590.584227] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1604.031943] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1612.311199] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.29@tcp (stopping) [ 1614.817749] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1614.820911] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1614.847054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1615.853896] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1615.863548] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1615.883229] Lustre: Skipped 1 previous similar message [ 1618.755527] Lustre: server umount lustre-MDT0000 complete [ 1620.968414] LustreError: 40122:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1621.003608] LustreError: 40122:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1624.834873] LustreError: 40103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785452301 with bad export cookie 5593651981434459699 [ 1624.844828] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1624.852523] LustreError: 40103:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1626.081270] LustreError: 40118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1626.081675] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1626.099037] LustreError: 40118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1626.116633] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1626.116640] Lustre: Skipped 3 previous similar messages [ 1632.789857] Lustre: lustre-MDT0001: Not available for connect from 192.168.203.29@tcp (stopping) [ 1632.808819] Lustre: Skipped 2 previous similar messages [ 1634.276709] LustreError: 40118:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1634.296973] LustreError: 40118:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 1639.392122] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1640.143238] Lustre: server umount lustre-MDT0001 complete [ 1656.960945] Lustre: server umount lustre-OST0000 complete [ 1683.704628] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1690.479693] Key type lgssc unregistered [ 1691.159587] LNet: 43992:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1691.177693] LNetError: 43992:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1691.248077] LNet: Removed LNI 192.168.203.129@tcp [ 1693.838109] Key type .llcrypt unregistered [ 1693.839960] Key type ._llcrypt unregistered [ 1728.441249] Key type ._llcrypt registered [ 1728.445680] Key type .llcrypt registered [ 1728.825317] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1731.031437] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1731.344083] alg: No test for adler32 (adler32-zlib) [ 1732.797312] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 1733.277271] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1735.104149] Key type lgssc registered [ 1737.849377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1749.610397] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1749.655894] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1751.157328] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1756.082513] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1760.740689] LustreError: 45412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1760.760197] LustreError: 45412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1765.860659] LustreError: 45413:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1767.361403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1768.065604] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1775.518989] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1786.439045] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1794.085934] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1805.178368] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1805.994607] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1816.341302] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1820.553533] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1829.141679] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1839.076289] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1839.098534] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1839.127938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1841.130523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1841.132012] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1841.135694] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1846.248293] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1846.273528] Lustre: Skipped 4 previous similar messages [ 1851.360412] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1851.363731] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1851.395913] Lustre: Skipped 1 previous similar message [ 1851.792886] Lustre: server umount lustre-OST0000 complete [ 1875.425471] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1875.436837] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1875.467587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1875.475392] Lustre: Skipped 1 previous similar message [ 1881.685837] Lustre: server umount lustre-MDT0000 complete [ 1882.095537] LustreError: 45409:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1882.129277] LustreError: 45409:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1887.202904] LustreError: 45412:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1887.238000] LustreError: 45412:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 1888.778261] LustreError: 45395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785452565 with bad export cookie 2070657264794671596 [ 1888.793174] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1888.796267] LustreError: 45395:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1895.635129] Lustre: server umount lustre-MDT0001 complete [ 1911.486672] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 1914.897319] Key type lgssc unregistered [ 1915.273757] LNet: 49143:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1915.278839] LNetError: 49143:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1915.305194] LNet: Removed LNI 192.168.203.129@tcp [ 1916.559569] Key type .llcrypt unregistered [ 1916.564482] Key type ._llcrypt unregistered [ 1949.074809] Key type ._llcrypt registered [ 1949.080355] Key type .llcrypt registered [ 1949.239484] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 1950.996826] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1951.047313] alg: No test for adler32 (adler32-zlib) [ 1952.117474] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 1952.466928] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 1954.240413] Key type lgssc registered [ 1955.552347] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1965.223390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1965.262142] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1966.796229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1972.776800] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 1977.342403] LustreError: 50560:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1977.411455] LustreError: 50560:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 1982.457173] LustreError: 50561:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1984.535535] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1985.662520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1993.172256] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2004.556441] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2015.519106] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2025.337886] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2025.701495] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2034.057199] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2036.965423] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 2044.175910] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2052.433376] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2056.971080] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 2070.068693] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-07-3019h05m43s [ 2072.022152] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 2077.194034] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.29@tcp (stopping) [ 2077.667963] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2077.683735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2077.718457] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2078.822269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2078.829328] Lustre: Skipped 5 previous similar messages [ 2080.841694] Lustre: server umount lustre-MDT0000 complete [ 2084.335633] LustreError: 51997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2085.553281] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 2090.476980] LustreError: 50555:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2090.510692] LustreError: 50555:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 2093.645275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.804412] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2094.227261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2094.316874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 2095.243482] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2100.512916] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2105.429282] Lustre: DEBUG MARKER: Wait for df (36799) ... [ 2108.108937] Lustre: DEBUG MARKER: done [ 2116.069523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2116.080367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2116.110125] Lustre: Skipped 3 previous similar messages [ 2121.186789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2121.189553] Lustre: Skipped 2 previous similar messages [ 2122.087815] Lustre: server umount lustre-OST0000 complete [ 2131.426671] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2131.453343] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2131.461029] Lustre: Skipped 1 previous similar message [ 2134.259605] Lustre: server umount lustre-MDT0000 complete [ 2138.601677] LustreError: 51289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2138.637655] LustreError: 51289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2139.392801] LustreError: 50541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785452816 with bad export cookie 6578944183855523386 [ 2139.412646] LustreError: 50541:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2139.422068] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2140.498843] Lustre: server umount lustre-MDT0001 complete [ 2154.164347] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2157.379582] Key type lgssc unregistered [ 2157.748122] LNet: 55492:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2157.759522] LNetError: 55492:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2157.774269] LNet: Removed LNI 192.168.203.129@tcp [ 2158.915928] Key type .llcrypt unregistered [ 2158.917512] Key type ._llcrypt unregistered [ 2210.826636] Key type ._llcrypt registered [ 2210.828836] Key type .llcrypt registered [ 2211.015567] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2212.826696] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2212.875633] alg: No test for adler32 (adler32-zlib) [ 2214.037664] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 2214.428561] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2216.224403] Key type lgssc registered [ 2217.192851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2226.525553] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2226.572264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2227.993774] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2228.032592] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2228.288187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2233.893982] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2245.707429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2245.951087] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2246.034417] Lustre: 57704:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2246.377489] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2252.445072] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2264.009224] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2271.176540] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2278.933449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2279.156494] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2279.431363] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2287.409790] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2287.842473] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 2300.100434] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2308.843592] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-07-3019h09m42s [ 2311.226299] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 2321.913364] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 2325.091826] Lustre: DEBUG MARKER: Creating a test file and stat it [ 2325.170529] LustreError: 58652:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 2345.526264] Lustre: lustre-MDT0000: Client 2982879f-de3b-464e-8022-9daeea63f954 (at 192.168.203.29@tcp) reconnecting [ 2365.209614] LustreError: 58652:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 2368.276059] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 2371.369612] Lustre: DEBUG MARKER: done [ 2384.870235] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2384.886561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2384.910536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2387.299406] Lustre: server umount lustre-OST0000 complete [ 2395.109990] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2395.132483] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2395.150463] Lustre: Skipped 1 previous similar message [ 2395.157111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2395.171226] Lustre: Skipped 2 previous similar messages [ 2396.273244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2396.283413] Lustre: Skipped 6 previous similar messages [ 2402.787178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2406.368158] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2407.347944] Lustre: server umount lustre-MDT0000 complete [ 2413.113045] LustreError: 57696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785453089 with bad export cookie 1094622117482836342 [ 2413.114313] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2413.127857] LustreError: 57696:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2413.787483] Lustre: server umount lustre-MDT0001 complete [ 2435.233099] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2438.993176] Key type lgssc unregistered [ 2439.442616] LNet: 62080:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2439.453128] LNetError: 62080:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2439.507779] LNet: Removed LNI 192.168.203.129@tcp [ 2440.767698] Key type .llcrypt unregistered [ 2440.768944] Key type ._llcrypt unregistered [ 2509.150246] Key type ._llcrypt registered [ 2509.153610] Key type .llcrypt registered [ 2509.321911] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2510.700832] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2510.801494] alg: No test for adler32 (adler32-zlib) [ 2512.198850] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 2512.769328] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2514.600920] Key type lgssc registered [ 2516.650576] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2521.725402] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2521.770856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2523.447532] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2523.505250] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2523.779216] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2525.358855] Lustre: Failing over lustre-MDT0000 [ 2526.221081] Lustre: server umount lustre-MDT0000 complete [ 2536.345322] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:13:30 (1785453210) [ 2545.483068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2546.047876] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2551.951610] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2563.393906] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2563.538664] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2563.568213] Lustre: 64883:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2563.936933] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2569.315648] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2580.214783] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2590.500224] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2600.036957] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2600.342418] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2600.623462] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2606.792966] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2609.190697] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2619.504099] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2627.322536] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2637.806076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2637.839592] Lustre: Skipped 1 previous similar message [ 2637.846855] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2637.881666] Lustre: Skipped 2 previous similar messages [ 2639.867139] Lustre: server umount lustre-MDT0000 complete [ 2645.544795] LustreError: 64875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785453322 with bad export cookie 1414623879411400437 [ 2645.568270] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2645.570122] LustreError: 64875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2646.611211] Lustre: server umount lustre-MDT0001 complete [ 2653.369566] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2660.740896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2663.393738] Lustre: 63806:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785453324/real 1785453324] req@000000003c7aed6d x1872183363154048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785453340 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2663.442190] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2663.471198] Lustre: Skipped 1 previous similar message [ 2665.441647] Lustre: 63805:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785453326/real 1785453326] req@0000000083787819 x1872183363154176/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785453342 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2668.512614] Lustre: 63803:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785453329/real 1785453329] req@000000004c72f508 x1872183363154304/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785453345 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2671.590170] Lustre: 63804:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785453332/real 1785453332] req@00000000b14b9565 x1872183363154560/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785453348 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2675.515113] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2676.704607] Lustre: 63806:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785453337/real 1785453337] req@000000000044bc08 x1872183363154688/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785453353 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2691.753059] LustreError: 66732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2691.782869] LustreError: 66732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2691.889825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2692.025353] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2692.917915] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2692.944432] LustreError: 66733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.949793] LustreError: 66732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2697.976910] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2700.270731] LustreError: 66733:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2705.381264] LustreError: 66732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2710.467931] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2710.948554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2716.142260] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2716.290967] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2726.403288] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2734.626644] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2746.850639] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2746.853989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2746.854697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2746.880766] Lustre: Skipped 2 previous similar messages [ 2746.915429] Lustre: Skipped 2 previous similar messages [ 2751.973091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2751.982585] Lustre: Skipped 2 previous similar messages [ 2752.934586] Lustre: server umount lustre-MDT0000 complete [ 2757.108459] LustreError: 69313:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2757.132044] LustreError: 69313:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 2757.736858] LustreError: 66738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785453434 with bad export cookie 1414623879411402712 [ 2757.747114] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2757.758519] LustreError: 66738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2758.518458] Lustre: server umount lustre-MDT0001 complete [ 2768.187268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2777.568648] Lustre: 63805:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785453438/real 1785453438] req@000000007ff1e4ba x1872183363186048/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1785453454 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2777.627057] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2783.240816] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2802.543374] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x13a1c0ec5f1a0d3b [ 2802.585972] Lustre: MGC192.168.203.129@tcp: Connection restored to 0@lo (at 0@lo) [ 2803.022078] LustreError: 69398:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2803.173379] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2803.279034] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2809.508375] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2812.336099] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2822.373688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2828.269128] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2828.823758] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 2840.176431] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2848.236780] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2864.106102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2864.123908] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2864.144915] Lustre: Skipped 1 previous similar message [ 2864.152150] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2864.162096] Lustre: Skipped 1 previous similar message [ 2865.986765] Lustre: server umount lustre-MDT0000 complete [ 2869.224916] LustreError: 72839:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2869.271940] LustreError: 72839:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 2870.779483] LustreError: 66738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785453547 with bad export cookie 1414623879411404091 [ 2870.793959] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2870.810025] LustreError: 66738:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2871.333132] Lustre: server umount lustre-MDT0001 complete [ 2879.160478] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2883.707363] Lustre: DEBUG MARKER: files compared the same [ 2897.263604] Lustre: server umount lustre-OST0000 complete [ 2926.483631] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 2931.678619] Key type lgssc unregistered [ 2932.077464] LNet: 74655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2932.088771] LNetError: 74655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2932.130691] LNet: Removed LNI 192.168.203.129@tcp [ 2933.892380] Key type .llcrypt unregistered [ 2933.896504] Key type ._llcrypt unregistered [ 2969.313476] Key type ._llcrypt registered [ 2969.315602] Key type .llcrypt registered [ 2969.429137] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 2971.369257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2971.442714] alg: No test for adler32 (adler32-zlib) [ 2972.736951] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 2973.208786] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 2975.040321] Key type lgssc registered [ 2977.246929] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2986.980685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2987.017351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2988.584106] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2994.725629] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug +malloc all [ 2998.753826] LustreError: 76077:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2998.778883] LustreError: 76077:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3003.875920] LustreError: 76078:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3005.726279] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3006.169579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3012.185795] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug +malloc all [ 3020.918463] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3028.636242] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3037.935160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3038.347535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3046.823739] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug +malloc all [ 3056.311779] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3058.794265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 3074.528870] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3074.534772] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3074.547134] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3074.563858] Lustre: Skipped 1 previous similar message [ 3074.583172] Lustre: Skipped 1 previous similar message [ 3079.703486] Lustre: server umount lustre-OST0000 complete [ 3084.775282] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3084.789122] Lustre: Skipped 1 previous similar message [ 3084.792977] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3084.812372] Lustre: Skipped 1 previous similar message [ 3089.889269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3089.902104] Lustre: Skipped 1 previous similar message [ 3091.512218] Lustre: server umount lustre-MDT0000 complete [ 3095.009825] LustreError: 76806:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3095.044863] LustreError: 76806:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3097.886711] LustreError: 76058:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785453774 with bad export cookie 17866959199981425935 [ 3097.910921] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3098.821246] Lustre: server umount lustre-MDT0001 complete [ 3118.148169] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 3121.470659] Key type lgssc unregistered [ 3121.942702] LNet: 79661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3121.983249] LNetError: 79661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3123.046936] LNet: Removed LNI 192.168.203.129@tcp [ 3124.496769] Key type .llcrypt unregistered [ 3124.498312] Key type ._llcrypt unregistered [ 3158.115822] Key type ._llcrypt registered [ 3158.125295] Key type .llcrypt registered [ 3158.420978] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3161.932540] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3161.973451] alg: No test for adler32 (adler32-zlib) [ 3163.329201] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 3163.971499] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 3165.992183] Key type lgssc registered [ 3169.191044] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3181.178370] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3181.206157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3196.896444] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 3202.017401] LustreError: 81054:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [ 3228.710066] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3236.315520] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3246.683784] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3257.490627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3257.784081] Lustre: *** cfs_fail_loc=706, val=0*** [ 3258.584419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3258.729254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 3263.722753] LustreError: 81080:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3263.747358] LustreError: 81080:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3264.450901] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3269.088917] LustreError: 82150:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3269.109791] LustreError: 82150:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3274.220094] LustreError: 82148:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3276.691382] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3277.259987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3283.141314] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3293.178180] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3301.358234] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3304.929352] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3304.939865] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3304.962054] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3308.001153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3308.007798] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3308.009871] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3308.055690] Lustre: Skipped 1 previous similar message [ 3309.921753] Lustre: server umount lustre-OST0000 complete [ 3318.241677] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3318.264350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3318.276044] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3318.312045] Lustre: Skipped 1 previous similar message [ 3322.192426] Lustre: server umount lustre-MDT0000 complete [ 3323.365552] LustreError: 83847:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3323.425037] LustreError: 83847:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3327.532662] LustreError: 82153:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3327.572507] LustreError: 82153:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3328.811126] LustreError: 81061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785454005 with bad export cookie 285639509928463668 [ 3328.831877] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3328.833269] LustreError: 81061:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3329.472701] Lustre: server umount lustre-MDT0001 complete [ 3348.904554] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 3353.775348] Key type lgssc unregistered [ 3354.241740] LNet: 84703:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3354.259635] LNetError: 84703:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3354.284802] LNet: Removed LNI 192.168.203.129@tcp [ 3355.776482] Key type .llcrypt unregistered [ 3355.781544] Key type ._llcrypt unregistered [ 3391.329268] Key type ._llcrypt registered [ 3391.341584] Key type .llcrypt registered [ 3391.663258] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3393.784116] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3393.921802] alg: No test for adler32 (adler32-zlib) [ 3395.924254] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 3396.795322] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 3398.828941] Key type lgssc registered [ 3401.838905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3416.922184] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3416.943856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3424.835502] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3436.228607] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3437.432987] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3442.661297] LustreError: 86813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3442.675161] LustreError: 86813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 3444.408409] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3446.754052] LustreError: 86812:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3446.786794] LustreError: 86812:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3451.877231] LustreError: 86813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3451.921540] LustreError: 86813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3456.997179] LustreError: 86816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3457.014134] LustreError: 86816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3457.783755] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3458.465736] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3461.483108] LustreError: 86813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3461.533204] LustreError: 86813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3468.284299] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3471.846075] LustreError: 86816:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3471.873424] LustreError: 86816:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [ 3480.805830] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3480.824959] Lustre: MGS: Client 44705c32-63c5-45d8-a33b-3f41e66691f8 (at 0@lo) reconnecting [ 3480.884777] LustreError: 86827:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3481.362458] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3481.485402] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 3489.401262] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3507.200511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3507.214970] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3507.235156] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3508.705201] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3508.716792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3508.739287] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3512.295425] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3513.710156] Lustre: server umount lustre-OST0000 complete [ 3522.530225] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3522.530609] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3522.583038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3522.589271] Lustre: Skipped 2 previous similar messages [ 3524.919639] Lustre: server umount lustre-MDT0000 complete [ 3527.649968] LustreError: 86813:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.675731] LustreError: 86813:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 3530.725682] Lustre: server umount lustre-MDT0001 complete [ 3535.478326] Lustre: server umount MGS complete [ 3552.634820] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:30:26 (1785454226) [ 3603.985864] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 3619.431427] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3667.646804] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3674.267991] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3685.770575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3686.088389] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3686.144916] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3686.222640] Lustre: lustre-MDT0001: new disk, initializing [ 3686.331194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3686.369241] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3686.386122] Lustre: 94026:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3686.398190] LustreError: 94026:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 1, retries 0, failed: rc = -115 [ 3691.980586] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3707.927835] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3708.463088] Lustre: lustre-OST0000: new disk, initializing [ 3708.473444] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3708.500763] Lustre: 94792:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3708.642915] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3716.713878] Lustre: 95006:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@0000000094ad0242 x1872184290096512/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785454409 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3718.742791] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3731.603312] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3737.056831] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3740.225104] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3740.242795] Lustre: MGS: Client 1ada5344-3377-4733-b84d-64c7406402db (at 0@lo) reconnecting [ 3740.335397] Lustre: 93296:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3740.375320] LustreError: 94012:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3740.398199] LustreError: 94012:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3740.420386] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3740.583840] Lustre: lustre-MDT0000: new disk, initializing [ 3740.767765] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3740.804917] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3741.860053] Lustre: 95863:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@000000000c9ed11a x1872184290104064/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1785454434 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3746.678176] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3746.984285] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3751.099765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3751.112257] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3751.156502] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 3776.993029] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3776.998465] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3777.014302] Lustre: Skipped 1 previous similar message [ 3777.021019] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3786.212602] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3786.225682] Lustre: Skipped 3 previous similar messages [ 3790.304225] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3790.797465] Lustre: server umount lustre-OST0000 complete [ 3796.451551] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3796.453542] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3796.475858] LustreError: Skipped 1 previous similar message [ 3796.518637] Lustre: Skipped 2 previous similar messages [ 3802.054509] Lustre: server umount lustre-MDT0000 complete [ 3806.690193] LustreError: 96216:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3806.729575] LustreError: 96216:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 3814.249667] Lustre: server umount lustre-MDT0001 complete [ 3819.963258] Lustre: server umount MGS complete [ 3838.883335] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:35:13 (1785454513) [ 3867.396791] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 3872.563680] Key type lgssc unregistered [ 3872.994908] LNet: 98386:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3873.017208] LNetError: 98386:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3873.032574] LNet: Removed LNI 192.168.203.129@tcp [ 3874.303981] Key type .llcrypt unregistered [ 3874.307292] Key type ._llcrypt unregistered [ 3896.610934] Key type ._llcrypt registered [ 3896.613833] Key type .llcrypt registered [ 3896.747384] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 3898.353193] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3898.415417] alg: No test for adler32 (adler32-zlib) [ 3899.398347] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 3899.659580] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 3901.392126] Key type lgssc registered [ 3903.389603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3914.970403] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3914.997021] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3916.551851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3923.323537] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3927.017273] LustreError: 99609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3927.037608] LustreError: 99609:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 3928.038537] LustreError: 99610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3933.155262] LustreError: 99609:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3933.621686] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3934.046626] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3939.618720] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3949.004366] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3957.126117] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3968.055685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3968.598176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3978.696476] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 3989.114567] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:33) [ 3989.696244] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4004.839853] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4004.844215] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4004.871553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4004.902435] LustreError: Skipped 1 previous similar message [ 4008.816477] Lustre: server umount lustre-OST0000 complete [ 4014.358494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4014.362585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.396817] Lustre: Skipped 2 previous similar messages [ 4014.428822] Lustre: Skipped 2 previous similar messages [ 4019.684752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4021.242498] Lustre: server umount lustre-MDT0000 complete [ 4024.808474] LustreError: 99610:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4024.827971] LustreError: 99610:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4026.655091] LustreError: 99591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785454703 with bad export cookie 5115241395434215286 [ 4026.670676] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4026.671335] LustreError: 99591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4027.744869] Lustre: server umount lustre-MDT0001 complete [ 4042.884516] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 4046.332601] Key type lgssc unregistered [ 4046.748639] LNet: 103144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4046.760667] LNetError: 103144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4046.799195] LNet: Removed LNI 192.168.203.129@tcp [ 4048.149772] Key type .llcrypt unregistered [ 4048.153502] Key type ._llcrypt unregistered [ 4074.843860] Key type ._llcrypt registered [ 4074.857122] Key type .llcrypt registered [ 4075.155215] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 4077.778936] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4077.827775] alg: No test for adler32 (adler32-zlib) [ 4079.190371] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 4079.893671] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 4081.768170] Key type lgssc registered [ 4083.936467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4094.464115] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4094.466967] LustreError: 104232:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 4094.477746] LustreError: 104233:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 4094.485271] LustreError: 104232:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4985 [ 4094.491662] LustreError: 104232:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 4094.493883] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4094.497589] LustreError: 104232:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4096.042536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4105.520924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4105.971378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4112.391838] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4120.409589] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing lsmod [ 4124.649919] LustreError: 105438:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 4124.655345] LustreError: 105441:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 4124.669648] LustreError: 105438:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 4124.679075] LustreError: 105438:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 4124.690582] LustreError: 105438:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 4124.704028] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4125.375942] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4130.785799] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4130.790082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4130.811857] Lustre: Skipped 2 previous similar messages [ 4130.814247] Lustre: Skipped 1 previous similar message [ 4131.878732] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4131.889153] Lustre: Skipped 3 previous similar messages [ 4135.924104] Lustre: server umount lustre-MDT0000 complete [ 4136.929927] LustreError: 104271:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4136.953516] LustreError: 104271:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [ 4142.051739] LustreError: 105323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4142.087397] LustreError: 105323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4142.257347] LustreError: 104244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785454818 with bad export cookie 13732454023356808553 [ 4142.269695] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4142.289726] LustreError: 104244:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4144.226094] LustreError: 104702:0:(client.c:1381:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000000c6639c9 x1872185006251008/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4144.918528] Lustre: server umount lustre-MDT0001 complete [ 4160.266543] Lustre: server umount lustre-OST0000 complete [ 4173.322302] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4174.129909] LustreError: 106508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4174.279948] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4179.434626] LustreError: 106509:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4181.029202] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4183.527178] LustreError: 106508:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4192.224152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4192.688965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4199.143826] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4210.992456] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4221.413763] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4233.971816] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4234.761062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4243.965118] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:3 to 0x240000400:65) [ 4245.149496] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4258.065761] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4269.538486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4269.550647] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4269.553843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4269.574996] Lustre: Skipped 1 previous similar message [ 4269.592731] Lustre: Skipped 1 previous similar message [ 4274.659112] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4274.678510] Lustre: Skipped 1 previous similar message [ 4275.439108] Lustre: server umount lustre-OST0000 complete [ 4284.900119] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4284.900167] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4284.965964] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4284.981864] Lustre: Skipped 1 previous similar message [ 4287.068427] Lustre: server umount lustre-MDT0000 complete [ 4290.022097] LustreError: 109003:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4290.036906] LustreError: 109003:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4292.056182] LustreError: 106489:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785454968 with bad export cookie 13732454023356809253 [ 4292.071192] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4293.196662] Lustre: server umount lustre-MDT0001 complete [ 4308.401808] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 4312.829896] Key type lgssc unregistered [ 4313.305066] LNet: 110048:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4313.311088] LNetError: 110048:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4313.337137] LNet: Removed LNI 192.168.203.129@tcp [ 4314.637962] Key type .llcrypt unregistered [ 4314.642080] Key type ._llcrypt unregistered [ 4352.250404] Key type ._llcrypt registered [ 4352.256699] Key type .llcrypt registered [ 4352.402359] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 4354.516802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4354.589099] alg: No test for adler32 (adler32-zlib) [ 4355.619765] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 4355.939891] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 4357.744879] Key type lgssc registered [ 4359.510709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4369.075890] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4369.112525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4370.796197] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4376.926863] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4381.154297] LustreError: 111471:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4381.191261] LustreError: 111471:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4386.294206] LustreError: 111470:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4389.962338] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4390.596486] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4396.890132] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4406.850125] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4415.112254] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4424.552795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4424.935927] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4433.858737] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4441.350390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:67 to 0x240000400:97) [ 4444.366620] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4451.688343] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 4457.447973] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 4467.180517] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4467.185237] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4467.197839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4467.227353] Lustre: Skipped 1 previous similar message [ 4471.434848] Lustre: server umount lustre-OST0000 complete [ 4477.302927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4477.332431] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4477.345114] Lustre: Skipped 1 previous similar message [ 4477.363338] Lustre: Skipped 2 previous similar messages [ 4482.561114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4482.571199] Lustre: Skipped 1 previous similar message [ 4484.063932] Lustre: server umount lustre-MDT0000 complete [ 4486.693449] LustreError: 111465:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4486.749825] LustreError: 111465:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4491.411063] LustreError: 111451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785455168 with bad export cookie 2357882656356178925 [ 4491.415532] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4491.422232] LustreError: 111451:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4491.639411] Lustre: server umount lustre-MDT0001 complete [ 4513.124972] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 4517.022386] Key type lgssc unregistered [ 4517.540679] LNet: 115138:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4517.556516] LNetError: 115138:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4517.603502] LNet: Removed LNI 192.168.203.129@tcp [ 4518.771559] Key type .llcrypt unregistered [ 4518.776081] Key type ._llcrypt unregistered [ 4544.277779] Key type ._llcrypt registered [ 4544.284379] Key type .llcrypt registered [ 4544.482404] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 4546.353778] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4546.385990] alg: No test for adler32 (adler32-zlib) [ 4547.784725] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 4548.252308] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 4550.010471] Key type lgssc registered [ 4552.033483] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4561.907459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4561.949859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4563.610330] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4570.655922] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4574.183303] LustreError: 116360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4574.217332] LustreError: 116360:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4579.300230] LustreError: 116360:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4584.465328] LustreError: 116361:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4586.259729] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4586.989619] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4593.590403] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4606.419317] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4614.970374] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4625.807900] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4626.488866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4633.665649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:99 to 0x240000400:129) [ 4634.708528] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4644.855862] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4657.122050] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4657.141982] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4657.168476] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4659.169449] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4659.174961] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4659.185546] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4659.217212] Lustre: Skipped 1 previous similar message [ 4662.350751] Lustre: server umount lustre-OST0000 complete [ 4668.897428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4668.917401] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4668.938215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4673.476751] Lustre: server umount lustre-MDT0000 complete [ 4674.556607] LustreError: 117602:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4674.578210] LustreError: 117602:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4678.128146] LustreError: 116341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785455354 with bad export cookie 11990324687120993045 [ 4678.139334] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4678.141726] LustreError: 116341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4678.654762] Lustre: server umount lustre-MDT0001 complete [ 4692.757858] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 4696.016868] Key type lgssc unregistered [ 4696.360231] LNet: 119900:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4696.373602] LNetError: 119900:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4696.398913] LNet: Removed LNI 192.168.203.129@tcp [ 4697.771581] Key type .llcrypt unregistered [ 4697.775440] Key type ._llcrypt unregistered [ 4729.424556] Key type ._llcrypt registered [ 4729.431977] Key type .llcrypt registered [ 4729.559550] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 4731.183786] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4731.271107] alg: No test for adler32 (adler32-zlib) [ 4732.404151] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 4732.709747] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 4734.344107] Key type lgssc registered [ 4735.851655] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4745.489491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4745.521813] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4747.247187] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4753.922056] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4756.452801] LustreError: 121319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4756.471902] LustreError: 121319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 4761.569087] LustreError: 121318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4766.691156] LustreError: 121319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4767.888655] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4768.473107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4773.978538] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4784.712349] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4793.091043] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4803.176196] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4803.716301] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4812.071510] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 4817.039385] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:131 to 0x240000400:161) [ 4823.015301] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4831.758601] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 4836.278571] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 4836.287317] Lustre: Skipped 2 previous similar messages [ 4839.500914] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 4839.539429] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 4841.389253] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 4841.396872] Lustre: Skipped 2 previous similar messages [ 4846.691470] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4846.706948] Lustre: Skipped 1 previous similar message [ 4853.626555] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.203.29@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4853.638906] Lustre: Skipped 2 previous similar messages [ 4855.521830] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.203.29@tcp 192.168.0.[2,10]@tcp [ 4866.427167] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.203.29/24@tcp in log lustre-MDT0000 [ 4866.440171] Lustre: Skipped 2 previous similar messages [ 4873.915761] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.29/24@tcp [ 4873.927835] Lustre: Skipped 1 previous similar message [ 4883.940427] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4883.947989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4883.962763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4884.973240] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4884.982273] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4885.003883] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4887.010897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4890.215795] Lustre: server umount lustre-OST0000 complete [ 4896.737736] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4896.754350] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4896.765356] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4896.768120] Lustre: Skipped 1 previous similar message [ 4902.386073] LustreError: 121318:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4902.426929] LustreError: 121318:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4902.618239] Lustre: server umount lustre-MDT0000 complete [ 4907.505043] LustreError: 121315:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4907.539019] LustreError: 121315:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 4909.304904] LustreError: 122053:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785455585 with bad export cookie 16895841902541623313 [ 4909.324521] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4911.132479] Lustre: server umount lustre-MDT0001 complete [ 4932.776074] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 4937.057389] Key type lgssc unregistered [ 4937.490614] LNet: 126247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4937.501354] LNetError: 126247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4937.519877] LNet: Removed LNI 192.168.203.129@tcp [ 4938.817072] Key type .llcrypt unregistered [ 4938.820586] Key type ._llcrypt unregistered [ 4976.242706] Key type ._llcrypt registered [ 4976.244894] Key type .llcrypt registered [ 4976.347688] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 4978.560876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4978.631501] alg: No test for adler32 (adler32-zlib) [ 4980.421769] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 4981.434055] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 4983.224209] Key type lgssc registered [ 4985.533833] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4996.881483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4996.920078] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4998.625969] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4998.654845] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.203.29/24@tcp [ 4998.688693] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5004.762595] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5008.881375] LustreError: 127810:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5008.902815] LustreError: 127810:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 5014.007842] LustreError: 127811:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5015.053891] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5015.535097] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 5015.566551] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.203.29/24@tcp [ 5015.633645] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5020.367573] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5031.058288] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5039.225223] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5049.542502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5050.005490] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5058.146853] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5066.940535] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5068.520029] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000400:169 to 0x240000400:193) [ 5087.203564] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5087.233873] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5087.266553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5089.259571] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5089.261240] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5091.203576] Lustre: server umount lustre-OST0000 complete [ 5099.489553] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5099.519400] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5099.535816] Lustre: Skipped 2 previous similar messages [ 5104.265404] Lustre: server umount lustre-MDT0000 complete [ 5104.618084] LustreError: 129240:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5108.723325] LustreError: 127792:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785455785 with bad export cookie 3269601414536149510 [ 5108.740268] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5109.460259] Lustre: server umount lustre-MDT0001 complete [ 5125.769357] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 5129.766452] Key type lgssc unregistered [ 5130.319954] LNet: 131401:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5130.328259] LNetError: 131401:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5130.379243] LNet: Removed LNI 192.168.203.129@tcp [ 5131.673672] Key type .llcrypt unregistered [ 5131.678574] Key type ._llcrypt unregistered [ 5181.327952] Key type ._llcrypt registered [ 5181.329437] Key type .llcrypt registered [ 5181.476069] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 5201.133270] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 5202.508670] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5202.779762] alg: No test for adler32 (adler32-zlib) [ 5204.211654] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 5204.630166] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 5206.456174] Key type lgssc registered [ 5208.220301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5255.853111] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5255.934618] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5257.335505] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5257.392187] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5257.558816] Lustre: lustre-MDT0000: new disk, initializing [ 5257.705172] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5257.734057] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5264.198155] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5281.747263] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5281.990864] Lustre: 134807:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5282.050583] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5282.056340] Lustre: Skipped 1 previous similar message [ 5282.120644] Lustre: lustre-MDT0001: new disk, initializing [ 5282.190745] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5282.221417] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5282.231938] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5289.367644] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5301.442972] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5309.840234] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5319.309554] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5319.948599] Lustre: lustre-OST0000: new disk, initializing [ 5319.964986] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5319.982382] Lustre: 136780:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5320.183360] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5326.959234] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5326.975229] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5327.115993] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5329.883729] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5344.450734] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5350.704874] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5350.934176] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5356.984670] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5357.221986] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5363.173704] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5363.176285] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5363.219092] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5363.228170] Lustre: Skipped 1 previous similar message [ 5365.928734] Lustre: server umount lustre-OST0000 complete [ 5373.411406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5373.417319] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5373.433351] Lustre: Skipped 2 previous similar messages [ 5373.455835] Lustre: Skipped 1 previous similar message [ 5378.534589] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5378.543690] Lustre: Skipped 1 previous similar message [ 5382.627277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5382.639062] Lustre: Skipped 1 previous similar message [ 5385.187349] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5385.820368] Lustre: server umount lustre-MDT0000 complete [ 5391.318984] LustreError: 134800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785456067 with bad export cookie 3744284747245688152 [ 5391.327340] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5391.355500] LustreError: 134800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5392.933484] Lustre: server umount lustre-MDT0001 complete [ 5406.433036] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5406.987813] LustreError: 139015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5407.020442] LustreError: 139015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5407.065089] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5412.329894] LustreError: 139016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5412.362810] LustreError: 139016:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5412.916478] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5416.426044] LustreError: 139015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5421.541695] LustreError: 139016:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5424.148794] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5424.665100] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5430.410203] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5440.796472] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5449.399433] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5459.747600] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5460.106550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5469.447970] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5480.020353] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5486.977737] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5487.515460] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5494.468476] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5494.815897] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5507.305819] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5507.555468] Lustre: lustre-OST0001: new disk, initializing [ 5507.563251] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5507.578254] Lustre: 142390:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5507.693612] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5516.195286] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5516.865195] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5516.881109] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5516.992497] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5529.684211] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5530.085591] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5537.041228] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5537.412590] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5546.828620] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff984ac6db6800.ost_server_uuid 50 [ 5549.136876] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff984ac6db6800.ost_server_uuid in IDLE state after 0 sec [ 5568.484359] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5568.509615] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5568.531312] LustreError: Skipped 1 previous similar message [ 5568.533558] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5568.558395] Lustre: Skipped 1 previous similar message [ 5570.367963] Lustre: server umount lustre-OST0001 complete [ 5573.622421] LustreError: 142710:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5573.660237] LustreError: 142710:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5578.211371] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5578.224061] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5578.238914] Lustre: Skipped 1 previous similar message [ 5578.248202] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5582.494697] Lustre: server umount lustre-OST0000 complete [ 5588.459946] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5588.481115] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5588.507557] Lustre: Skipped 1 previous similar message [ 5594.346584] Lustre: server umount lustre-MDT0000 complete [ 5599.204604] LustreError: 139015:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5599.235654] LustreError: 139015:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5599.424081] LustreError: 142389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785456276 with bad export cookie 3744284747245689034 [ 5599.439672] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5599.449065] LustreError: 142389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5600.286832] Lustre: server umount lustre-MDT0001 complete [ 5626.414606] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 5630.195278] Key type lgssc unregistered [ 5630.662243] LNet: 145421:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5630.671752] LNetError: 145421:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5631.721856] LNet: Removed LNI 192.168.203.129@tcp [ 5632.905888] Key type .llcrypt unregistered [ 5632.909885] Key type ._llcrypt unregistered [ 5687.258993] Key type ._llcrypt registered [ 5687.261914] Key type .llcrypt registered [ 5687.407486] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 5706.980331] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 5708.351166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5708.458328] alg: No test for adler32 (adler32-zlib) [ 5709.915402] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 5710.312859] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 5712.120164] Key type lgssc registered [ 5713.933650] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5767.719459] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5767.789381] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5769.481972] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5769.522367] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5769.807380] Lustre: lustre-MDT0000: new disk, initializing [ 5769.973521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5770.037380] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5777.598718] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5791.417626] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5791.596686] Lustre: 149581:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5791.654749] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5791.660410] Lustre: Skipped 1 previous similar message [ 5791.709584] Lustre: lustre-MDT0001: new disk, initializing [ 5791.836641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5791.877510] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5791.890527] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5797.410522] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5811.792556] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5821.498987] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5834.758644] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5835.164772] Lustre: lustre-OST0000: new disk, initializing [ 5835.175074] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5835.185250] Lustre: 151553:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5835.306944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5843.578444] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5844.627297] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5844.641086] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5844.794531] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5859.103977] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5869.010020] Lustre: Failing over lustre-OST0000 [ 5869.446817] Lustre: server umount lustre-OST0000 complete [ 5886.951834] Lustre: 147802:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785456547/real 1785456547] req@0000000029651f96 x1872186715446272/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785456563 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5886.956995] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5887.008672] Lustre: 147802:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5892.065726] Lustre: 147801:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785456552/real 1785456552] req@00000000b747eca4 x1872186715448192/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785456568 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5892.125332] Lustre: 147801:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5895.784798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5896.318105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5896.357615] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 5897.315763] Lustre: 147801:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785456557/real 1785456557] req@000000004bea9908 x1872186715449344/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785456573 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5897.333479] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 5897.354097] Lustre: 147801:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5897.768639] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5897.772537] Lustre: lustre-OST0000: Recovery over after 0:01, of 3 clients 3 recovered and 0 were evicted. [ 5901.793081] Lustre: 147802:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785456562/real 1785456562] req@0000000010d17a06 x1872186715450368/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1785456578 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5901.862996] Lustre: 147802:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 5904.975912] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5910.085130] Lustre: Failing over lustre-MDT0000 [ 5911.026836] Lustre: server umount lustre-MDT0000 complete [ 5913.061106] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5913.077281] Lustre: Skipped 3 previous similar messages [ 5918.179207] LustreError: 149588:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5918.211491] LustreError: 149588:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [ 5923.304160] LustreError: 149595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5923.343330] LustreError: 149595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5928.417365] Lustre: 147803:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785456589/real 1785456589] req@000000002849251f x1872186715462912/t0(0) o400->MGC192.168.203.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785456605 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5928.424051] LustreError: 149594:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5928.444355] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5928.487744] LustreError: 149594:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5932.462836] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5933.540635] LustreError: 149595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5933.577574] LustreError: 149595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5938.673355] LustreError: 149591:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5938.677046] LustreError: 147800:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005d7df43d x1872186715467904/t0(0) o250->MGC192.168.203.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5938.697598] LustreError: 149591:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 5939.047599] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5939.166627] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5939.700846] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5943.800832] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 5944.306831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5944.325810] Lustre: Skipped 1 previous similar message [ 5944.380828] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5944.458084] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5959.137025] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5959.149302] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5959.168136] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5962.359442] Lustre: server umount lustre-OST0000 complete [ 5969.889876] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5969.914218] Lustre: Skipped 1 previous similar message [ 5969.940348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5969.942918] Lustre: Skipped 1 previous similar message [ 5973.826939] Lustre: server umount lustre-MDT0000 complete [ 5975.010417] LustreError: 149595:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5975.026822] LustreError: 149595:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 5978.423867] LustreError: 149573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785456655 with bad export cookie 7749286440473311108 [ 5978.429673] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5978.443987] LustreError: 149573:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5978.822838] Lustre: server umount lustre-MDT0001 complete [ 5994.825113] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 5998.861194] Key type lgssc unregistered [ 5999.290558] LNet: 155225:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5999.298610] LNetError: 155225:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5999.341246] LNet: Removed LNI 192.168.203.129@tcp [ 6000.723559] Key type .llcrypt unregistered [ 6000.733362] Key type ._llcrypt unregistered [ 6045.017468] Key type ._llcrypt registered [ 6045.023657] Key type .llcrypt registered [ 6045.158704] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 6046.851928] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6047.132469] alg: No test for adler32 (adler32-zlib) [ 6048.293885] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 6048.689352] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 6050.480612] Key type lgssc registered [ 6051.933734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6061.898597] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6061.942376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6063.854425] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6070.715971] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6074.337553] LustreError: 156748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6074.353938] LustreError: 156748:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6079.457634] LustreError: 156747:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6083.190386] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6084.585402] LustreError: 156748:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6087.764687] LustreError: 156776:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6087.793201] LustreError: 156776:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6087.805655] LustreError: 156776:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 24, retries 0, failed: rc = -5 [ 6088.407636] Lustre: server umount lustre-MDT0000 complete [ 6111.813234] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 6124.261219] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 6178.242950] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.753158] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6178.812964] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6179.141813] Lustre: lustre-MDT0000: new disk, initializing [ 6179.263145] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6179.283995] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6187.962525] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6204.121856] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6204.261618] Lustre: 160085:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6204.335924] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6204.341766] Lustre: Skipped 1 previous similar message [ 6204.504607] Lustre: lustre-MDT0001: new disk, initializing [ 6204.706469] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6204.778711] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6204.782117] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6210.177753] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6222.552581] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6230.612769] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6239.595686] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6240.096744] Lustre: lustre-OST0000: new disk, initializing [ 6240.106703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6240.120586] Lustre: 162058:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6240.279676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6247.837345] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6248.142881] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6248.149989] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6248.355174] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6260.226342] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6265.860914] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 6268.707454] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 20:15:42 (1785456942) [ 6588.897502] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6588.905652] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6588.926067] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6591.458421] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6591.459078] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6591.488937] Lustre: Skipped 2 previous similar messages [ 6593.782418] Lustre: server umount lustre-MDT0000 complete [ 6596.593509] LustreError: 160096:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6596.619205] LustreError: 160096:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6598.460898] LustreError: 160076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785457275 with bad export cookie 15161684754745900464 [ 6598.464035] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6598.488889] LustreError: 160076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6601.697306] LustreError: 160093:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6601.702533] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6601.742362] Lustre: Skipped 1 previous similar message [ 6601.751160] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6605.364546] Lustre: server umount lustre-MDT0001 complete [ 6625.248114] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6625.638931] Lustre: server umount lustre-OST0000 complete [ 6637.147253] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 6650.027700] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 6698.565784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6699.002524] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6699.058515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6699.265549] Lustre: lustre-MDT0000: new disk, initializing [ 6699.409063] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6699.436158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6704.641501] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6716.761426] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6716.927749] Lustre: 166736:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6717.048627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6717.055460] Lustre: Skipped 1 previous similar message [ 6717.225706] Lustre: lustre-MDT0001: new disk, initializing [ 6717.391113] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6717.474948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6717.501830] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6723.462819] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6738.727139] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6745.610385] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6755.152873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6755.671961] Lustre: lustre-OST0000: new disk, initializing [ 6755.678526] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6755.683388] Lustre: 168707:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6755.803438] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6765.566797] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6765.807224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6765.870826] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6766.096076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6783.905900] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6792.540089] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6792.946329] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6799.734347] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6800.003101] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6807.009671] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6807.031174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6807.063021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6807.075863] Lustre: Skipped 2 previous similar messages [ 6808.642769] Lustre: server umount lustre-OST0000 complete [ 6814.689987] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6814.701275] LustreError: Skipped 1 previous similar message [ 6814.710224] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6814.731975] Lustre: Skipped 1 previous similar message [ 6814.741543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6819.126095] Lustre: server umount lustre-MDT0000 complete [ 6822.370031] LustreError: 166742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6822.392151] LustreError: 166742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 6824.228026] LustreError: 166727:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785457500 with bad export cookie 15161684754746059301 [ 6824.250929] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6825.217424] Lustre: server umount lustre-MDT0001 complete [ 6836.831690] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:25:11 (1785457511) [ 6865.211258] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 6876.756640] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 6920.578797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6920.873669] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6920.903331] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6921.044882] Lustre: lustre-MDT0000: new disk, initializing [ 6921.135666] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6921.159384] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6925.993780] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6939.258380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6939.423654] Lustre: 173471:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6939.973579] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6948.866990] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 6963.669583] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6972.664613] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6984.367832] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6984.912474] Lustre: lustre-OST0000: new disk, initializing [ 6984.918727] Lustre: Skipped 1 previous similar message [ 6984.926760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6984.933886] Lustre: Skipped 2 previous similar messages [ 6984.940383] Lustre: 175451:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6993.543248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6993.556354] Lustre: Skipped 1 previous similar message [ 6993.570413] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6993.707987] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6994.448191] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7006.615587] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7019.309683] Lustre: server umount lustre-OST0000 complete [ 7024.612737] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7024.639235] Lustre: Skipped 1 previous similar message [ 7024.658253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7024.666354] Lustre: Skipped 2 previous similar messages [ 7034.079482] Lustre: server umount lustre-MDT0000 complete [ 7034.853232] LustreError: 173478:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7034.886366] LustreError: 173478:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7035.877105] Lustre: 156058:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785457696/real 1785457696] req@00000000a2c06a6f x1872187071685632/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1785457712 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 7035.904435] Lustre: 156058:0:(client.c:2490:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 7038.559785] LustreError: 173464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785457715 with bad export cookie 15161684754746060183 [ 7038.573420] LustreError: 173464:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7038.582129] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7039.094766] Lustre: server umount lustre-MDT0001 complete [ 7049.679292] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 20:28:44 (1785457724) [ 7069.043825] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 7082.972607] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7131.182294] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7131.567552] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7131.616377] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7131.785709] Lustre: lustre-MDT0000: new disk, initializing [ 7131.918743] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7131.929386] Lustre: Skipped 2 previous similar messages [ 7131.970661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7138.252890] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7153.647039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7153.828772] Lustre: 179775:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7154.293063] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7161.100918] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7174.750256] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7183.245270] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7194.122235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7194.391132] Lustre: 181755:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7203.945917] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7204.138376] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7205.232950] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7218.495255] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7229.924527] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7229.925432] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7229.939348] LustreError: Skipped 1 previous similar message [ 7229.940763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7229.940769] Lustre: Skipped 2 previous similar messages [ 7229.995799] Lustre: Skipped 3 previous similar messages [ 7234.912329] Lustre: server umount lustre-OST0000 complete [ 7241.185188] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7246.120703] Lustre: server umount lustre-MDT0000 complete [ 7248.359545] LustreError: 179780:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7248.401300] LustreError: 179780:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 7251.012852] LustreError: 181753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785457927 with bad export cookie 15161684754746061268 [ 7251.024469] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7251.039538] LustreError: 181753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7251.893492] Lustre: server umount lustre-MDT0001 complete [ 7274.083345] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 7278.712410] Key type lgssc unregistered [ 7279.310660] LNet: 183637:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7279.328422] LNetError: 183637:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7279.389236] LNet: Removed LNI 192.168.203.129@tcp [ 7281.122541] Key type .llcrypt unregistered [ 7281.126924] Key type ._llcrypt unregistered [ 7314.474747] Key type ._llcrypt registered [ 7314.478829] Key type .llcrypt registered [ 7314.622906] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7315.978413] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7316.261507] alg: No test for adler32 (adler32-zlib) [ 7317.397095] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 7317.955546] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 7319.720173] Key type lgssc registered [ 7322.249772] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7332.291189] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7332.325212] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7333.910905] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7339.729620] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7344.098187] LustreError: 185056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7344.121288] LustreError: 185056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7349.222430] LustreError: 185057:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7351.156222] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7351.594130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7357.953164] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7368.993859] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7378.608568] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7388.476406] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7389.208328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7399.225156] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7403.516829] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 7411.728038] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7434.215099] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7434.228876] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7434.232656] LustreError: Skipped 1 previous similar message [ 7434.235729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7434.257284] Lustre: Skipped 1 previous similar message [ 7435.788645] Lustre: server umount lustre-OST0000 complete [ 7443.937206] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7443.948423] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7443.978167] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7443.980738] Lustre: Skipped 1 previous similar message [ 7449.569217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7449.571754] Lustre: Skipped 3 previous similar messages [ 7453.669249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7455.201286] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7456.126274] Lustre: server umount lustre-MDT0000 complete [ 7458.787360] LustreError: 185056:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7458.822373] LustreError: 185056:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7462.877955] LustreError: 185037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785458139 with bad export cookie 12068723397998822263 [ 7462.895275] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7462.903538] LustreError: 185037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7463.800175] Lustre: server umount lustre-MDT0001 complete [ 7479.152514] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 7482.977485] Key type lgssc unregistered [ 7483.541300] LNet: 188615:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7483.551089] LNetError: 188615:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7483.596185] LNet: Removed LNI 192.168.203.129@tcp [ 7484.889278] Key type .llcrypt unregistered [ 7484.894923] Key type ._llcrypt unregistered [ 7517.783311] Key type ._llcrypt registered [ 7517.788210] Key type .llcrypt registered [ 7517.930783] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 7519.775517] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7519.801379] alg: No test for adler32 (adler32-zlib) [ 7521.679671] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 7522.369908] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 7524.328478] Key type lgssc registered [ 7526.218792] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7537.079197] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7537.136218] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7538.904343] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7545.085058] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7549.411869] LustreError: 190046:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7549.435246] LustreError: 190046:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 7554.530951] LustreError: 190047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7555.688409] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7556.277923] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7561.756275] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7571.544099] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7578.841944] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7586.178345] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7586.708583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7594.356443] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7603.216958] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7610.309249] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 7620.671349] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7620.683413] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7620.714574] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7620.780129] Lustre: Skipped 1 previous similar message [ 7625.740443] Lustre: lustre-OST0000: Not available for connect from 192.168.203.29@tcp (stopping) [ 7625.771338] Lustre: Skipped 4 previous similar messages [ 7630.821873] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7630.847106] Lustre: Skipped 1 previous similar message [ 7631.328121] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7631.796861] Lustre: server umount lustre-OST0000 complete [ 7646.575601] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff984ac4355000.ost_server_uuid 50 [ 7648.792724] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 7665.303294] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7665.310689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7665.334896] Lustre: Skipped 1 previous similar message [ 7665.356807] Lustre: Skipped 2 previous similar messages [ 7668.004652] Lustre: server umount lustre-MDT0000 complete [ 7670.754092] LustreError: 190047:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7670.770211] LustreError: 190047:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7672.510781] LustreError: 190026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785458349 with bad export cookie 16859010008655738659 [ 7672.521777] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7672.525440] LustreError: 190026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7673.244116] Lustre: server umount lustre-MDT0001 complete [ 7684.170422] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 20:39:18 (1785458358) [ 7692.182121] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7692.755521] LustreError: 193996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7692.876289] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7697.895617] LustreError: 193997:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7698.005761] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7708.131846] LustreError: 193996:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7708.156759] LustreError: 193996:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 7710.370237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7710.896464] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7717.729961] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7728.617597] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7736.960463] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7745.119969] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7745.813802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7752.880566] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7762.873196] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7763.258400] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7772.957160] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7773.319478] Lustre: lustre-OST0001: new disk, initializing [ 7773.330259] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7773.345196] Lustre: 196840:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7773.514249] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7782.070311] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7782.078034] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7782.328266] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7782.947243] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7794.990284] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7799.265438] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7799.269927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7799.297097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7802.338869] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7802.342761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7805.484480] Lustre: server umount lustre-OST0000 complete [ 7805.945342] LustreError: 195820:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7814.526492] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7814.900773] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7821.170580] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7821.458316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7838.689303] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7838.705786] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7838.706396] LustreError: 195821:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7838.718170] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7838.718178] Lustre: Skipped 3 previous similar messages [ 7838.739697] Lustre: Skipped 1 previous similar message [ 7838.814700] LustreError: 195821:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 18 previous similar messages [ 7839.635238] Lustre: server umount lustre-OST0001 complete [ 7848.944718] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7848.945716] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7848.979790] Lustre: Skipped 1 previous similar message [ 7849.016337] LustreError: Skipped 1 previous similar message [ 7851.249309] Lustre: server umount lustre-MDT0000 complete [ 7856.400022] LustreError: 193976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785458533 with bad export cookie 16859010008655739730 [ 7856.414454] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7856.431630] LustreError: 193976:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7857.180751] Lustre: server umount lustre-MDT0001 complete [ 7888.038783] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 20:42:42 (1785458562) [ 7897.198627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7897.771696] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7897.878511] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7898.301201] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7904.353669] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7916.758640] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7916.874396] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7916.967060] Lustre: 200281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7923.923353] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7935.162535] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7946.594535] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7956.473062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7956.863543] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7967.465930] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 7968.279932] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 7979.462756] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7991.330181] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7991.472145] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7991.828124] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7991.835427] Lustre: Skipped 2 previous similar messages [ 8000.129929] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8000.354541] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 8012.938141] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8018.401391] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8018.406540] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8018.431029] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8018.436378] Lustre: Skipped 3 previous similar messages [ 8022.563922] Lustre: server umount lustre-OST0000 complete [ 8024.057354] LustreError: 202480:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8024.084645] LustreError: 202480:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 8041.452421] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8041.460358] LustreError: Skipped 2 previous similar messages [ 8043.872510] Lustre: server umount lustre-OST0001 complete [ 8051.170279] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8051.193761] Lustre: Skipped 3 previous similar messages [ 8051.203718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8051.208483] Lustre: Skipped 6 previous similar messages [ 8055.391655] Lustre: server umount lustre-MDT0000 complete [ 8060.074344] LustreError: 200275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785458736 with bad export cookie 16859010008655741137 [ 8060.075300] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8060.085334] LustreError: 200275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8060.678540] Lustre: server umount lustre-MDT0001 complete [ 8093.379898] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 20:46:07 (1785458767) [ 8118.800623] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 8134.566562] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 8193.836968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8194.255584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8194.320405] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8194.440461] Lustre: lustre-MDT0000: new disk, initializing [ 8194.504317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8194.543932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8200.503653] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8217.367632] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8217.563582] Lustre: 208393:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8217.670215] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8217.677593] Lustre: Skipped 1 previous similar message [ 8217.816994] Lustre: lustre-MDT0001: new disk, initializing [ 8218.052704] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8218.058295] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8226.017461] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8239.211662] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8248.806374] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8258.063813] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8258.548487] Lustre: lustre-OST0000: new disk, initializing [ 8258.566466] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8258.588584] Lustre: 210364:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8266.341379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8266.357069] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8266.617285] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8268.721336] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8283.476370] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8289.311978] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8289.481200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8294.901490] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8295.131308] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8302.592275] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8302.600806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8302.602701] LustreError: Skipped 2 previous similar messages [ 8302.603410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8302.603413] Lustre: Skipped 1 previous similar message [ 8302.658339] Lustre: Skipped 2 previous similar messages [ 8304.476175] Lustre: server umount lustre-OST0000 complete [ 8318.753531] Lustre: server umount lustre-MDT0000 complete [ 8323.051414] LustreError: 210934:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8323.076073] LustreError: 210934:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 13 previous similar messages [ 8323.495199] LustreError: 208385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785459000 with bad export cookie 16859010008655742922 [ 8323.499104] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8323.511878] LustreError: 208385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8323.992169] Lustre: server umount lustre-MDT0001 complete [ 8335.013111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8341.987234] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8355.687087] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8361.157669] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8372.674667] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8381.495752] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8391.316789] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8402.619565] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8416.279425] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8423.914324] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8424.394781] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8432.366547] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8432.732755] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8437.218096] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8437.221110] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8437.249377] Lustre: Skipped 2 previous similar messages [ 8437.264593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8437.281179] Lustre: Skipped 5 previous similar messages [ 8441.295113] Lustre: server umount lustre-OST0000 complete [ 8449.651249] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8455.005798] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8487.407090] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8487.638076] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8487.651074] Lustre: Skipped 5 previous similar messages [ 8489.721366] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8489.766541] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8496.308464] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8505.136688] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8505.174650] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8510.039740] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8514.197527] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8531.340081] Lustre: server umount lustre-OST0000 complete [ 8545.788610] Lustre: server umount lustre-MDT0000 complete [ 8551.593140] LustreError: 213323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785459228 with bad export cookie 16859010008655743804 [ 8551.605718] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8551.627183] LustreError: 213323:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8552.406107] Lustre: server umount lustre-MDT0001 complete [ 8565.418621] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 20:53:59 (1785459239) [ 8576.057280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8581.607409] LustreError: 218564:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8581.643384] LustreError: 218564:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 12 previous similar messages [ 8583.476322] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8594.837665] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8601.906747] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8611.380661] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8619.793053] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8630.947695] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8641.422924] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8654.440661] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8662.334178] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8662.586803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8669.292321] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8669.694621] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8679.484742] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8679.764355] Lustre: lustre-OST0001: new disk, initializing [ 8679.775660] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8679.791246] Lustre: 221941:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8688.348365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8688.368469] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8688.519071] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8689.857333] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8703.790250] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8713.641516] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8714.255163] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8721.175062] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8721.554557] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8724.449142] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8724.451936] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8724.457381] LustreError: Skipped 4 previous similar messages [ 8724.461968] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8724.488310] Lustre: Skipped 6 previous similar messages [ 8724.520575] Lustre: Skipped 8 previous similar messages [ 8738.272299] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8738.360066] Lustre: server umount lustre-OST0001 complete [ 8748.075649] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 8748.462527] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 8754.606988] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 8755.016491] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 8786.739874] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8792.551885] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8792.580165] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 8792.581874] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8792.601498] Lustre: Skipped 1 previous similar message [ 8797.834346] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8811.066589] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8813.672716] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8817.980615] Lustre: server umount lustre-OST0001 complete [ 8849.618064] Lustre: server umount lustre-OST0000 complete [ 8861.351030] Lustre: server umount lustre-MDT0000 complete [ 8866.913229] LustreError: 218544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785459543 with bad export cookie 16859010008655744770 [ 8866.941340] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8866.945020] LustreError: 218544:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8873.744917] Lustre: server umount lustre-MDT0001 complete [ 8904.836325] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 20:59:38 (1785459578) [ 8914.997982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8915.574691] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8915.651948] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8922.314269] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8933.637986] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8933.754364] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8933.817122] Lustre: 227195:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8938.777140] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8948.771312] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8955.650852] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8964.433037] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8964.809813] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8973.193652] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 8984.301423] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8997.774952] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8997.943920] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9008.000819] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9018.643990] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9021.055419] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9028.771766] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 9029.158342] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9036.391207] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 9036.774756] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9046.124586] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff984acd9f3800.ost_server_uuid 50 [ 9048.753684] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff984acd9f3800.ost_server_uuid in IDLE state after 0 sec [ 9050.001111] Lustre: Permanently deactivating lustre-OST0001 [ 9050.015572] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 9057.108346] Lustre: Permanently reactivating lustre-OST0001 [ 9057.151770] Lustre: Modifying parameter lustre-OST0001-osc.osc.active=1 in log lustre-client [ 9057.160177] Lustre: Skipped 2 previous similar messages [ 9063.908246] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9063.923187] LustreError: Skipped 4 previous similar messages [ 9070.136102] Lustre: server umount lustre-OST0001 complete [ 9082.304323] Lustre: server umount lustre-OST0000 complete [ 9103.328157] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9104.037700] Lustre: server umount lustre-MDT0000 complete [ 9104.880943] LustreError: 230708:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9104.925328] LustreError: 230708:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 52 previous similar messages [ 9109.793543] LustreError: 227188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785459786 with bad export cookie 16859010008655746058 [ 9109.818328] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9109.819962] LustreError: 227188:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9111.017175] Lustre: server umount lustre-MDT0001 complete [ 9152.199960] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 21:03:46 (1785459826) [ 9163.555500] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9164.227592] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9164.296835] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9164.329267] Lustre: Skipped 2 previous similar messages [ 9164.966983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9164.984821] Lustre: Skipped 9 previous similar messages [ 9173.599176] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9184.811446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9184.943486] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9184.991485] Lustre: 233916:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9191.585790] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9203.407436] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9212.641118] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9223.273137] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9223.458170] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9223.503502] Lustre: Permanently deactivating lustre-OST0000 [ 9223.520470] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9233.233315] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9233.462022] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9247.332768] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9257.740221] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9257.786897] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9264.864515] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 9265.572850] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9276.471996] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9278.731326] Lustre: Permanently reactivating lustre-OST0000 [ 9278.754745] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9278.768917] Lustre: Skipped 2 previous similar messages [ 9285.753522] Lustre: lustre-OST0000: Client 4a9f5f09-5ee8-4793-9e84-277aa88cbd09 (at 192.168.203.29@tcp) reconnecting [ 9285.758166] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0001-mdtlov_UUID (at 0@lo) in 52 seconds. I think it's dead, and I am evicting it. exp 00000000b3ac935e, cur 1785459962 deadline 1785459940 last 1785459910 [ 9286.828209] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 9300.975426] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9300.982152] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9300.990433] Lustre: Skipped 13 previous similar messages [ 9301.002666] Lustre: Skipped 24 previous similar messages [ 9311.209699] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9311.587551] Lustre: server umount lustre-OST0001 complete [ 9322.511305] Lustre: server umount lustre-OST0000 complete [ 9333.743698] Lustre: server umount lustre-MDT0000 complete [ 9338.550654] LustreError: 237513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785460015 with bad export cookie 16859010008655747738 [ 9338.573098] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9338.574199] LustreError: 237513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9339.432995] Lustre: server umount lustre-MDT0001 complete [ 9360.522690] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 9365.137749] Key type lgssc unregistered [ 9365.674728] LNet: 238825:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9365.686130] LNetError: 238825:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9365.704459] LNet: Removed LNI 192.168.203.129@tcp [ 9367.909740] Key type .llcrypt unregistered [ 9367.914300] Key type ._llcrypt unregistered [ 9404.067378] Key type ._llcrypt registered [ 9404.077881] Key type .llcrypt registered [ 9404.248532] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 9406.694128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9406.794375] alg: No test for adler32 (adler32-zlib) [ 9408.020803] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [ 9408.443071] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [ 9410.224298] Key type lgssc registered [ 9412.412836] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9435.276390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9435.335181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9437.702229] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9446.978546] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9447.907043] LustreError: 240296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9447.940735] LustreError: 240296:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [ 9453.034552] LustreError: 240297:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9458.150485] LustreError: 240296:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9461.431134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9461.624969] Lustre: Found index 1 for lustre-MDT0001, updating log [ 9461.688827] Lustre: 240286:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9461.705819] Lustre: Permanently deactivating lustre-MDT0001 [ 9461.742809] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9462.249386] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9467.434491] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9470.443120] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9482.304575] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9492.260435] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9503.858089] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9504.643421] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9506.733949] LustreError: 242145:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9506.779947] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 9507.772655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 9512.937715] LustreError: 242238:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9512.962687] LustreError: 242238:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9513.265313] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9523.175307] LustreError: 242486:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9523.209926] LustreError: 242486:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [ 9524.868683] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9533.900173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9534.170329] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9539.574377] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 9539.575341] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 9541.817139] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9551.968382] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9555.542300] Lustre: Permanently reactivating lustre-MDT0001 [ 9555.585060] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 9555.592375] Lustre: Skipped 2 previous similar messages [ 9559.215414] Lustre: lustre-MDT0001: Client 7361fe08-cb52-4084-81c9-e5aee2e4873b (at 192.168.203.29@tcp) reconnecting [ 9561.784608] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9561.806941] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 9561.819341] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 9561.836153] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 9567.042292] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9569.509658] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9579.272924] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 9579.697594] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 9586.034393] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 9586.434825] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 9590.369887] Lustre: Permanently deactivating lustre-MDT0001 [ 9590.422427] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 9590.424922] Lustre: Skipped 2 previous similar messages [ 9596.566201] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 9610.720961] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9610.735944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9610.762148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9613.541917] Lustre: server umount lustre-MDT0000 complete [ 9616.380263] LustreError: 241469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9616.402148] LustreError: 241469:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [ 9620.128351] LustreError: 240278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785460296 with bad export cookie 5631875928489059149 [ 9620.151430] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9620.159616] LustreError: 240278:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9621.070236] Lustre: server umount lustre-MDT0001 complete [ 9628.383126] Lustre: server umount lustre-OST0000 complete [ 9646.652467] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 21:12:00 (1785460320) [ 9662.537631] Lustre: server umount lustre-OST0001 complete [ 9673.291463] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [ 9688.142686] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [ 9741.504111] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9742.264119] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9742.297243] Lustre: Skipped 2 previous similar messages [ 9742.391782] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9742.740914] Lustre: lustre-MDT0000: new disk, initializing [ 9742.956701] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9743.009742] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9749.803308] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9767.996916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9768.190843] Lustre: 248594:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9768.337620] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9768.349315] Lustre: Skipped 1 previous similar message [ 9768.550072] Lustre: lustre-MDT0001: new disk, initializing [ 9768.736536] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9768.791982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9768.814052] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9779.571823] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9797.083445] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9807.920722] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9818.147293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9818.555856] Lustre: lustre-OST0000: new disk, initializing [ 9818.562934] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9818.576522] Lustre: 250564:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9818.711797] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9823.952331] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9823.969156] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9824.137058] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9826.977459] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9839.122350] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9842.310785] LustreError: 248602:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9852.314083] LustreError: 248602:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9852.350897] LustreError: 251152:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 9853.675037] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9853.962857] Lustre: lustre-OST0001: new disk, initializing [ 9853.968556] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9853.978968] Lustre: 251681:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9854.114513] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9861.762491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9861.774991] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9861.886753] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9862.401053] LustreError: 251152:0:(mdt_handler.c:3263:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 9862.475619] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [ 9875.072023] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9877.878778] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 9882.598993] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9882.600363] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9882.629284] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9882.629295] Lustre: Skipped 4 previous similar messages [ 9882.629525] Lustre: Skipped 4 previous similar messages [ 9887.714127] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9887.716478] Lustre: Skipped 1 previous similar message [ 9892.839189] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9892.850250] Lustre: Skipped 1 previous similar message [ 9895.392083] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9895.731258] Lustre: server umount lustre-OST0001 complete [ 9897.975245] LustreError: 250550:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9897.999374] LustreError: 250550:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [ 9918.448851] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9918.452888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9918.475088] Lustre: Skipped 1 previous similar message [ 9918.490661] Lustre: Skipped 1 previous similar message [ 9922.758213] Lustre: server umount lustre-OST0000 complete [ 9928.673269] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9928.702257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9928.720143] Lustre: Skipped 1 previous similar message [ 9934.484400] Lustre: server umount lustre-MDT0000 complete [ 9940.168182] LustreError: 248588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785460616 with bad export cookie 5631875928489061963 [ 9940.181059] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9940.186839] LustreError: 248588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9946.858841] Lustre: server umount lustre-MDT0001 complete [ 9963.966275] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [ 9967.871803] Key type lgssc unregistered [ 9968.462972] LNet: 253733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9968.474763] LNetError: 253733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9968.502657] LNet: Removed LNI 192.168.203.129@tcp [ 9970.230158] Key type .llcrypt unregistered [ 9970.237406] Key type ._llcrypt unregistered [10024.075694] Key type ._llcrypt registered [10024.081575] Key type .llcrypt registered [10024.240411] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [10026.283744] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10026.359811] alg: No test for adler32 (adler32-zlib) [10027.574625] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [10027.959325] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [10029.696181] Key type lgssc registered [10031.312749] Lustre: Echo OBD driver; http://www.lustre.org/ [10041.663461] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10041.695844] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10043.134694] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10043.196736] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10043.516133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10048.104209] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10058.981924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10059.151153] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10059.222463] Lustre: 255949:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10059.549387] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10059.800661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10066.427486] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10078.422136] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10089.110450] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10097.737308] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10098.193413] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10098.579755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10104.025104] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [10109.896159] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10123.731609] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10191.330600] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10191.333285] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10191.335680] LustreError: Skipped 1 previous similar message [10191.376423] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10191.382366] Lustre: Skipped 1 previous similar message [10191.401678] Lustre: Skipped 1 previous similar message [10193.016434] Lustre: server umount lustre-OST0000 complete [10199.248033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [10216.133986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10216.726578] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10225.172086] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10232.104596] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10232.147159] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [10232.147622] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [10232.197497] Lustre: Skipped 1 previous similar message [10239.761860] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10288.618880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10288.621949] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10288.923514] Lustre: server umount lustre-OST0000 complete [10298.851794] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10298.881988] Lustre: Skipped 1 previous similar message [10298.896266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10302.451372] Lustre: server umount lustre-MDT0000 complete [10303.970129] LustreError: 258548:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10304.017790] LustreError: 258548:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [10307.542262] LustreError: 257810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785460984 with bad export cookie 2115337537175876803 [10307.548484] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10307.554068] LustreError: 257810:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10308.078063] Lustre: server umount lustre-MDT0001 complete [10326.788069] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [10333.194609] Key type lgssc unregistered [10334.039438] LNet: 261404:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10334.042451] LNetError: 261404:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10334.099190] LNet: Removed LNI 192.168.203.129@tcp [10336.068111] Key type .llcrypt unregistered [10336.070293] Key type ._llcrypt unregistered [10377.381175] Key type ._llcrypt registered [10377.385151] Key type .llcrypt registered [10377.556717] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [10379.376356] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10379.424066] alg: No test for adler32 (adler32-zlib) [10381.255206] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [10381.730395] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [10383.568185] Key type lgssc registered [10384.724151] Lustre: Echo OBD driver; http://www.lustre.org/ [10395.359671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10395.399659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10397.307806] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10403.775246] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10406.882995] LustreError: 262824:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10406.906869] LustreError: 262824:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10412.001706] LustreError: 262825:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10415.267042] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10415.795046] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10424.665498] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10437.163569] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10445.471824] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10455.155382] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10455.776618] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10464.809340] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10467.082972] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [10477.720525] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10512.353179] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10512.369816] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10512.391299] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10513.380555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10513.381289] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10513.382587] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10513.398890] Lustre: Skipped 1 previous similar message [10517.079328] Lustre: server umount lustre-OST0000 complete [10523.617731] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10523.635922] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10523.639110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10523.659100] Lustre: Skipped 1 previous similar message [10528.741648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10528.766488] Lustre: Skipped 1 previous similar message [10533.858114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10533.866018] Lustre: Skipped 1 previous similar message [10535.392637] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10535.998679] Lustre: server umount lustre-MDT0000 complete [10538.977192] LustreError: 262819:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10538.993310] LustreError: 262819:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10541.141792] LustreError: 262807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785461217 with bad export cookie 18418075706582261510 [10541.152302] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10541.157569] LustreError: 262807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10541.663973] Lustre: server umount lustre-MDT0001 complete [10556.909586] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [10561.230980] Key type lgssc unregistered [10561.607707] LNet: 267139:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10561.617549] LNetError: 267139:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10561.627329] LNet: Removed LNI 192.168.203.129@tcp [10562.913325] Key type .llcrypt unregistered [10562.917203] Key type ._llcrypt unregistered [10587.001973] Key type ._llcrypt registered [10587.020570] Key type .llcrypt registered [10587.228223] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [10588.585298] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10588.776905] alg: No test for adler32 (adler32-zlib) [10590.164555] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [10590.625714] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [10592.440149] Key type lgssc registered [10594.376223] Lustre: Echo OBD driver; http://www.lustre.org/ [10608.803902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10608.851279] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10610.787206] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10616.978606] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10620.901287] LustreError: 268367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10620.927692] LustreError: 268367:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10626.081533] LustreError: 268368:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10630.773550] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10631.140885] LustreError: 268367:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10631.550758] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10638.636311] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10652.903343] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10661.922799] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10671.136302] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10671.577162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10678.959604] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10680.843405] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [10689.817979] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10706.402066] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10706.411217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10706.426217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10706.435370] Lustre: Skipped 1 previous similar message [10706.459127] Lustre: Skipped 1 previous similar message [10710.405806] Lustre: server umount lustre-OST0000 complete [10718.695697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10718.713503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10718.754792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10721.761706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10721.774159] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10723.541256] Lustre: server umount lustre-MDT0000 complete [10726.883658] LustreError: 269233:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10726.923839] LustreError: 269233:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10729.775907] LustreError: 270214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785461406 with bad export cookie 5123282072857623494 [10729.786330] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10729.802678] LustreError: 270214:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10730.367637] Lustre: server umount lustre-MDT0001 complete [10747.053929] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [10750.607556] Key type lgssc unregistered [10751.015474] LNet: 272051:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10751.019330] LNetError: 272051:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10751.037296] LNet: Removed LNI 192.168.203.129@tcp [10752.246542] Key type .llcrypt unregistered [10752.252941] Key type ._llcrypt unregistered [10786.063165] Key type ._llcrypt registered [10786.073562] Key type .llcrypt registered [10786.375475] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [10788.709484] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10788.738712] alg: No test for adler32 (adler32-zlib) [10790.168191] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [10790.559777] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [10792.385276] Key type lgssc registered [10793.505555] Lustre: Echo OBD driver; http://www.lustre.org/ [10801.677639] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10801.700233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10803.307757] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10808.633660] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10812.903813] LustreError: 273468:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10812.924943] LustreError: 273468:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [10818.018253] LustreError: 273469:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.848670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10819.328102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10824.873877] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10835.404683] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10844.906887] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10855.221759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10855.795887] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10863.451954] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [10873.211896] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [10873.307158] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10909.156026] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10909.158265] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10909.177794] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10909.194161] Lustre: Skipped 1 previous similar message [10912.066722] Lustre: server umount lustre-OST0000 complete [10919.395581] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10919.435026] Lustre: Skipped 1 previous similar message [10919.447287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10919.454899] Lustre: Skipped 1 previous similar message [10924.124873] Lustre: server umount lustre-MDT0000 complete [10924.513485] LustreError: 275311:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10924.552362] LustreError: 275311:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10928.618753] LustreError: 273464:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10928.642655] LustreError: 273464:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [10929.099047] LustreError: 273450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785461605 with bad export cookie 17618134416628131821 [10929.106210] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10929.124052] LustreError: 273450:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10930.163101] Lustre: server umount lustre-MDT0001 complete [10947.523512] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [10951.764349] Key type lgssc unregistered [10952.263042] LNet: 277781:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10952.285125] LNetError: 277781:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10952.311926] LNet: Removed LNI 192.168.203.129@tcp [10953.767032] Key type .llcrypt unregistered [10953.768463] Key type ._llcrypt unregistered [10977.476292] Key type ._llcrypt registered [10977.477891] Key type .llcrypt registered [10977.646292] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [10978.906980] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10978.947077] alg: No test for adler32 (adler32-zlib) [10980.258507] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [10980.652325] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [10982.392135] Key type lgssc registered [10984.020684] Lustre: Echo OBD driver; http://www.lustre.org/ [10993.546601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10993.570753] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10995.456566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11002.572916] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11004.907934] LustreError: 279020:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11004.940221] LustreError: 279020:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [11010.020679] LustreError: 279022:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11015.145876] LustreError: 279024:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11017.981140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11018.857884] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11027.162837] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11040.843164] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11049.389225] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11060.736991] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11061.167479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11065.301828] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [11071.180786] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11086.081765] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11105.252219] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11105.265299] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11105.285914] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11106.274302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11106.285832] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11106.295078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11110.607632] Lustre: server umount lustre-OST0000 complete [11116.390159] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11116.423671] Lustre: Skipped 1 previous similar message [11116.426277] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11116.440383] Lustre: Skipped 2 previous similar messages [11121.635224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11121.645206] Lustre: Skipped 1 previous similar message [11122.699522] Lustre: server umount lustre-MDT0000 complete [11126.764812] LustreError: 279032:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11126.789681] LustreError: 279032:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11126.966183] LustreError: 278985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785461803 with bad export cookie 9992441158345791628 [11126.986046] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11128.003510] Lustre: server umount lustre-MDT0001 complete [11148.314440] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [11153.178638] Key type lgssc unregistered [11153.539340] LNet: 282752:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11153.552047] LNetError: 282752:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11153.578092] LNet: Removed LNI 192.168.203.129@tcp [11154.813735] Key type .llcrypt unregistered [11154.817803] Key type ._llcrypt unregistered [11170.421537] Key type ._llcrypt registered [11170.434591] Key type .llcrypt registered [11170.565979] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [11199.095508] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [11219.234163] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [11220.393459] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11220.471357] alg: No test for adler32 (adler32-zlib) [11221.834358] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [11222.159091] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [11223.888520] Key type lgssc registered [11225.605840] Lustre: Echo OBD driver; http://www.lustre.org/ [11273.812089] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11273.885165] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11275.319538] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11275.395454] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11275.534483] Lustre: lustre-MDT0000: new disk, initializing [11275.684164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11275.730235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11281.673347] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11297.163685] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11297.340824] Lustre: 286316:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11297.460137] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11297.470372] Lustre: Skipped 1 previous similar message [11297.599497] Lustre: lustre-MDT0001: new disk, initializing [11297.767248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11297.871657] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11297.897781] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11306.176430] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11326.662786] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11337.321330] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11347.650876] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11348.432054] Lustre: lustre-OST0000: new disk, initializing [11348.444696] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11348.451973] Lustre: 288288:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11348.625270] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11358.289614] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11358.305417] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11358.529090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11361.661872] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11377.436590] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11384.058413] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11384.395293] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11391.601794] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11391.941792] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11399.649249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11399.653176] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11399.672872] LustreError: Skipped 1 previous similar message [11399.678269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11399.715812] Lustre: Skipped 1 previous similar message [11404.770148] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11404.772361] Lustre: Skipped 1 previous similar message [11409.376247] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11409.992486] Lustre: server umount lustre-OST0000 complete [11415.521303] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11415.548044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11415.583048] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11420.653678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11420.657126] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11420.682087] Lustre: Skipped 1 previous similar message [11422.485257] Lustre: server umount lustre-MDT0000 complete [11428.257458] LustreError: 287111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785462104 with bad export cookie 5510518108715899074 [11428.258855] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11428.268659] LustreError: 287111:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11429.029721] Lustre: server umount lustre-MDT0001 complete [11441.550337] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 21:41:56 (1785462116) [11452.020386] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11453.250296] LustreError: 290703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11453.287813] LustreError: 290703:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [11453.360090] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11458.532239] LustreError: 290704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11459.891955] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11462.638109] LustreError: 290703:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11467.749651] LustreError: 290704:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11472.202380] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11473.034249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11480.703312] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11492.608866] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11502.416945] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11517.508024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11518.144254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11529.417350] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11547.049584] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11568.609733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11568.612235] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11568.626535] Lustre: Skipped 1 previous similar message [11568.644872] Lustre: Skipped 1 previous similar message [11578.853846] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11578.864299] Lustre: Skipped 3 previous similar messages [11579.360081] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11579.892653] Lustre: server umount lustre-OST0000 complete [11585.506344] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11585.517991] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11599.330636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11599.333312] Lustre: Skipped 5 previous similar messages [11599.334952] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [11599.890951] Lustre: server umount lustre-MDT0000 complete [11604.452329] LustreError: 290700:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11604.492636] LustreError: 290700:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [11605.251198] LustreError: 290685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785462281 with bad export cookie 5510518108715899956 [11605.259098] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11605.265897] LustreError: 290685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11606.533891] Lustre: server umount lustre-MDT0001 complete [11624.978369] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [11628.908541] Key type lgssc unregistered [11629.362482] LNet: 294293:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11629.392163] LNetError: 294293:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11629.421670] LNet: Removed LNI 192.168.203.129@tcp [11631.156145] Key type .llcrypt unregistered [11631.163110] Key type ._llcrypt unregistered [11673.849781] Key type ._llcrypt registered [11673.855217] Key type .llcrypt registered [11674.068577] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [11695.592388] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [11697.333791] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11697.363935] alg: No test for adler32 (adler32-zlib) [11698.350864] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [11698.539508] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [11700.203445] Key type lgssc registered [11702.007787] Lustre: Echo OBD driver; http://www.lustre.org/ [11766.447649] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11766.492786] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11767.960164] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11768.013732] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11768.141768] Lustre: lustre-MDT0000: new disk, initializing [11768.266592] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11768.319119] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11774.628900] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11787.372739] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11787.534303] Lustre: 298125:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11787.609879] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11787.623795] Lustre: Skipped 1 previous similar message [11787.770768] Lustre: lustre-MDT0001: new disk, initializing [11788.058299] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11788.142864] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11788.164969] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11795.342204] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11810.893424] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11819.756662] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11828.917303] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11829.592506] Lustre: lustre-OST2710: new disk, initializing [11829.606810] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [11829.621842] Lustre: 300093:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11829.886284] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [11835.604641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [11835.625230] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [11836.047502] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [11841.533828] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11860.230073] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11871.807607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11872.143255] Lustre: lustre-OST03e8: new disk, initializing [11872.168681] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [11872.182217] Lustre: 301166:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11872.405436] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [11877.577148] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [11877.604047] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [11877.760199] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [11883.372215] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [11893.356918] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11904.159814] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [11904.518402] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11910.522047] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [11910.878412] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11918.277159] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [11918.634493] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11926.073262] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [11926.417792] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11934.178485] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11934.198194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11934.210457] Lustre: Skipped 2 previous similar messages [11934.229166] Lustre: Skipped 3 previous similar messages [11938.779742] Lustre: server umount lustre-MDT0000 complete [11943.431101] LustreError: 298919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785462620 with bad export cookie 14680496147945000097 [11943.452371] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11943.456190] LustreError: 298919:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11944.417051] LustreError: 298131:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11944.420911] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11944.435080] LustreError: 298131:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [11944.475022] Lustre: Skipped 2 previous similar messages [11944.667744] Lustre: server umount lustre-MDT0001 complete [11949.608247] Lustre: server umount lustre-OST2710 complete [11954.444646] Lustre: server umount lustre-OST03e8 complete [11978.846604] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [11997.750735] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [12049.796053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12050.257322] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12050.295788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12050.470843] Lustre: lustre-MDT0000: new disk, initializing [12050.643391] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12050.668187] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12058.385705] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12071.231367] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12071.389894] Lustre: 306441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12071.502973] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12071.509716] Lustre: Skipped 1 previous similar message [12071.646171] Lustre: lustre-MDT0001: new disk, initializing [12071.788420] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12071.838167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12071.854766] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12077.910986] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12092.570556] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12102.787073] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12112.788780] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12113.298156] Lustre: lustre-OST0000: new disk, initializing [12113.306425] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12113.319611] Lustre: 308409:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12113.444147] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12120.713053] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12120.729439] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12120.887161] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12122.815581] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12135.016418] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12142.005823] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12142.285737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12147.758308] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12148.038631] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12151.789171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12151.801367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12151.830032] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12156.605590] Lustre: server umount lustre-OST0000 complete [12164.065755] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12164.075582] LustreError: Skipped 1 previous similar message [12164.083306] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12164.103184] Lustre: Skipped 1 previous similar message [12164.113211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12164.130026] Lustre: Skipped 1 previous similar message [12167.141906] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12167.159966] Lustre: Skipped 1 previous similar message [12172.257629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12172.272219] Lustre: Skipped 1 previous similar message [12176.352817] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12176.908727] Lustre: server umount lustre-MDT0000 complete [12177.379597] LustreError: 307540:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12177.410184] LustreError: 307540:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [12181.741492] LustreError: 306433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785462858 with bad export cookie 14680496147945001791 [12181.751674] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12181.770417] LustreError: 306433:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12182.637803] Lustre: server umount lustre-MDT0001 complete [12193.468549] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 21:54:28 (1785462868) [12195.592423] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [12198.327361] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 21:54:32 (1785462872) [12202.907722] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [12249.288627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12249.734916] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12249.782371] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12250.044087] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12256.311489] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12268.197735] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12268.303060] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12268.349732] Lustre: 312275:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12275.929195] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12305.089220] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12305.589561] Lustre: Denying initial registration attempt from nid 192.168.203.129@tcp, specified as failover [12305.603994] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [12305.619557] LustreError: 313733:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [12305.636720] LustreError: 313733:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [12305.652185] LustreError: 313733:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [12305.807910] Lustre: server umount lustre-OST0000 complete [12305.810844] LustreError: 313733:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -99 [12319.718223] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12319.721744] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12319.734618] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12319.752766] Lustre: Skipped 2 previous similar messages [12319.771426] Lustre: Skipped 1 previous similar message [12321.939185] Lustre: server umount lustre-MDT0000 complete [12324.838682] LustreError: 312289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12324.869894] LustreError: 312289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12326.964464] LustreError: 313020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785463003 with bad export cookie 14680496147945002673 [12326.975808] LustreError: 313020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12326.978391] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12327.593119] Lustre: server umount lustre-MDT0001 complete [12344.401165] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12349.357413] Key type lgssc unregistered [12349.801466] LNet: 314855:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12349.810369] LNetError: 314855:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12350.824531] LNet: Removed LNI 192.168.203.129@tcp [12352.578492] Key type .llcrypt unregistered [12352.579967] Key type ._llcrypt unregistered [12367.993503] Key type ._llcrypt registered [12367.998429] Key type .llcrypt registered [12368.192616] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [12369.806318] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12370.086876] alg: No test for adler32 (adler32-zlib) [12371.427215] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [12371.875081] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [12373.648676] Key type lgssc registered [12375.123751] Lustre: Echo OBD driver; http://www.lustre.org/ [12433.316596] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [12446.718739] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12446.758755] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12448.345545] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12448.373096] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12448.630444] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12455.702496] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12467.707788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12467.872236] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12467.961379] Lustre: 317625:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12468.284727] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12474.046201] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12483.691729] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12492.551953] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12514.412657] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12514.719955] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12515.053134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12522.753837] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12534.070384] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12539.880207] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12539.886743] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12539.896144] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12539.905799] Lustre: Skipped 1 previous similar message [12539.946434] LustreError: Skipped 1 previous similar message [12545.010030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12545.019994] Lustre: Skipped 2 previous similar messages [12550.129661] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12550.139836] Lustre: Skipped 1 previous similar message [12551.138598] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12551.620844] Lustre: server umount lustre-OST0000 complete [12560.354496] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12560.356458] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12560.386704] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12560.423513] Lustre: Skipped 1 previous similar message [12563.039215] Lustre: server umount lustre-MDT0000 complete [12569.598953] LustreError: 319504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785463246 with bad export cookie 7254109008528394609 [12569.605494] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12569.613833] LustreError: 319504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12570.418726] Lustre: server umount lustre-MDT0001 complete [12587.575694] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12591.713336] Key type lgssc unregistered [12592.295497] LNet: 321218:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12592.323737] LNetError: 321218:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12592.349119] LNet: Removed LNI 192.168.203.129@tcp [12594.163319] Key type .llcrypt unregistered [12594.170385] Key type ._llcrypt unregistered [12631.718574] Key type ._llcrypt registered [12631.723908] Key type .llcrypt registered [12631.901989] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [12634.275273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12634.330868] alg: No test for adler32 (adler32-zlib) [12635.743187] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [12636.221860] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [12638.088156] Key type lgssc registered [12639.711633] Lustre: Echo OBD driver; http://www.lustre.org/ [12650.317546] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12650.369662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12652.146779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12658.500163] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12661.734512] LustreError: 322639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12661.763522] LustreError: 322639:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12666.852131] LustreError: 322638:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12671.813489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12671.974068] LustreError: 322639:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12672.319971] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12679.707139] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12693.646481] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12702.410701] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12715.696260] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12716.359486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12725.296670] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12735.826778] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12748.814962] Lustre: lustre-MDT0000: Not available for connect from 192.168.203.29@tcp (stopping) [12749.287549] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12749.302103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12749.332905] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12752.353191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12752.369508] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12752.404837] Lustre: Skipped 1 previous similar message [12755.606856] Lustre: server umount lustre-MDT0000 complete [12757.476559] LustreError: 322635:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12757.531733] LustreError: 322635:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12761.953349] LustreError: 322620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785463438 with bad export cookie 3819947308096320221 [12761.976782] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12761.986702] LustreError: 322620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12762.594169] LustreError: 323515:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12762.595118] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12762.612793] LustreError: 323515:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [12762.657883] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12762.660046] Lustre: Skipped 3 previous similar messages [12769.058685] Lustre: server umount lustre-MDT0001 complete [12778.670424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [12789.759851] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12806.624824] LustreError: 321947:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000008878cc5d x1872193977895808/t0(0) o250->MGC192.168.203.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12807.135246] LustreError: 324474:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12807.160754] LustreError: 324474:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12807.237808] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12807.340051] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [12812.975481] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12822.509533] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12823.839356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12824.384594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12829.678083] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12830.568564] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12841.298308] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12843.451333] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [12851.690367] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12853.965210] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [12863.457659] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12863.473435] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12863.523981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12863.532042] Lustre: Skipped 2 previous similar messages [12865.521513] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12869.094679] Lustre: server umount lustre-OST0000 complete [12875.746557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12875.747298] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12875.748394] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12875.748398] Lustre: Skipped 2 previous similar messages [12875.789584] Lustre: Skipped 2 previous similar messages [12881.831684] Lustre: server umount lustre-MDT0000 complete [12885.987470] LustreError: 326247:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12886.015173] LustreError: 326247:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [12887.452547] LustreError: 322620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785463564 with bad export cookie 3819947308096328061 [12887.472161] LustreError: 322620:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12887.483497] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12888.308874] Lustre: server umount lustre-MDT0001 complete [12905.304068] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [12910.162686] Key type lgssc unregistered [12910.570490] LNet: 328920:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12910.574294] LNetError: 328920:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12911.591103] LNet: Removed LNI 192.168.203.129@tcp [12912.943367] Key type .llcrypt unregistered [12912.944725] Key type ._llcrypt unregistered [12949.284228] Key type ._llcrypt registered [12949.286857] Key type .llcrypt registered [12949.455232] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [12951.503405] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12951.595339] alg: No test for adler32 (adler32-zlib) [12953.247346] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [12953.877397] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [12955.760166] Key type lgssc registered [12959.202756] Lustre: Echo OBD driver; http://www.lustre.org/ [12973.057779] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12973.085253] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12974.687050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12980.888317] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [12984.805641] LustreError: 330343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12984.826091] LustreError: 330343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [12989.955829] LustreError: 330344:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12992.161312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12992.523865] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12999.260657] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13009.327180] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13016.352917] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13023.201291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13023.214357] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13023.234142] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13025.250020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13025.256249] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13026.794395] Lustre: server umount lustre-MDT0000 complete [13030.371042] LustreError: 330343:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13030.394232] LustreError: 330343:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13030.758590] LustreError: 330325:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785463707 with bad export cookie 8866538687518955245 [13030.775416] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13031.197809] Lustre: server umount lustre-MDT0001 complete [13052.009118] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13052.840555] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [13052.898163] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13053.135219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13059.876481] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13082.945466] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13083.217376] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [13083.321380] Lustre: 332932:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13083.912602] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13090.696994] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13102.411943] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13112.552589] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13129.318549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13129.527873] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [13129.560597] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [13129.572874] LustreError: 334820:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [13129.579672] LustreError: 334820:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [13129.587817] LustreError: 334820:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [13129.695566] Lustre: server umount lustre-OST0000 complete [13129.704810] LustreError: 334820:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -2 [13146.040147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13146.609860] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [13147.117727] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13152.947088] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [13155.502603] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13166.905752] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13178.830065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13179.008326] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [13179.205507] Lustre: lustre-OST0001: new disk, initializing [13179.216322] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [13179.227946] Lustre: 336365:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13179.345379] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [13184.643242] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [13184.668161] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [13184.777844] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [13190.454409] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13206.890943] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13210.593867] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13210.595369] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13210.602447] LustreError: Skipped 1 previous similar message [13210.609251] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [13210.609259] Lustre: Skipped 1 previous similar message [13210.645244] Lustre: Skipped 1 previous similar message [13215.724770] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [13215.727196] Lustre: Skipped 1 previous similar message [13216.632750] Lustre: server umount lustre-OST0001 complete [13220.854985] LustreError: 336994:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13220.880935] LustreError: 336994:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13224.419293] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13224.434769] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13224.457710] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13226.005105] LustreError: 335287:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13226.024274] LustreError: 335287:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [13235.168431] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13235.603632] Lustre: server umount lustre-OST0000 complete [13241.320307] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13241.349114] Lustre: Skipped 1 previous similar message [13241.358148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13241.370429] Lustre: Skipped 5 previous similar messages [13247.194686] Lustre: server umount lustre-MDT0000 complete [13251.553976] LustreError: 334389:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13251.585342] LustreError: 334389:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13253.243050] LustreError: 336364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785463929 with bad export cookie 8866538687518955798 [13253.246241] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13253.266046] LustreError: 336364:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [13254.385943] Lustre: server umount lustre-MDT0001 complete [13272.473764] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [13275.958745] Key type lgssc unregistered [13276.320239] LNet: 338342:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13276.324394] LNetError: 338342:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13276.345108] LNet: Removed LNI 192.168.203.129@tcp [13277.422840] Key type .llcrypt unregistered [13277.425192] Key type ._llcrypt unregistered [13369.653379] Key type ._llcrypt registered [13369.656906] Key type .llcrypt registered [13369.821778] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [13393.984258] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [13395.503290] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13395.551527] alg: No test for adler32 (adler32-zlib) [13396.853714] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [13397.198074] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [13398.995486] Key type lgssc registered [13401.095351] Lustre: Echo OBD driver; http://www.lustre.org/ [13452.651233] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13452.718968] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13454.126364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13454.213847] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13454.410112] Lustre: lustre-MDT0000: new disk, initializing [13454.615855] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13454.667577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13462.340873] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13476.807705] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13477.017727] Lustre: 343284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13477.166397] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13477.177263] Lustre: Skipped 1 previous similar message [13477.355412] Lustre: lustre-MDT0001: new disk, initializing [13477.472839] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13477.526784] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13477.541825] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13484.673393] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13498.494606] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13507.447820] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13516.825797] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13517.337642] Lustre: lustre-OST0000: new disk, initializing [13517.350600] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13517.363241] Lustre: 345255:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13517.536488] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13525.118041] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13525.142225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13525.328365] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13526.112239] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13540.766485] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13547.361614] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13547.679712] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13554.427478] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13554.746074] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13561.313621] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13561.317423] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13561.323098] LustreError: Skipped 1 previous similar message [13561.324236] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13561.354942] Lustre: Skipped 1 previous similar message [13563.891687] Lustre: server umount lustre-OST0000 complete [13571.555261] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13571.559278] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13571.585749] Lustre: Skipped 1 previous similar message [13571.620117] Lustre: Skipped 2 previous similar messages [13577.296851] Lustre: server umount lustre-MDT0000 complete [13581.754400] LustreError: 344081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785464258 with bad export cookie 699040579664025605 [13581.767648] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13581.782652] LustreError: 344081:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13582.192139] Lustre: server umount lustre-MDT0001 complete [13596.796638] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 22:17:50 (1785464270) [13608.826204] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 22:18:03 (1785464283) [13619.279243] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13620.054688] LustreError: 348017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13620.082086] LustreError: 348017:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [13620.197835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13625.316376] LustreError: 348018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13627.487546] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13629.413617] LustreError: 348017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13634.535057] LustreError: 348018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13639.651893] LustreError: 348017:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13640.224835] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13640.859454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13647.387678] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13657.954857] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13666.594581] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13675.515465] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13676.053606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13685.100896] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13697.088442] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13707.195880] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [13710.497994] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [13712.561763] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [13714.946279] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [13718.065272] Lustre: Failing over lustre-MDT0000 [13718.920377] Lustre: server umount lustre-MDT0000 complete [13720.108988] LustreError: 348013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13721.071539] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13721.091949] Lustre: Skipped 1 previous similar message [13736.417684] Lustre: 341504:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785464397/real 1785464397] req@0000000019addacc x1872194775893248/t0(0) o400->MGC192.168.203.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785464413 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13736.436561] LustreError: 348018:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13736.473201] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13736.558781] LustreError: 348018:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 15 previous similar messages [13745.547136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13746.663557] LustreError: 341502:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a3335a07 x1872194775898752/t0(0) o250->MGC192.168.203.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13747.468171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13747.603440] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13751.798497] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13752.821712] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13752.863904] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [13752.942795] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13753.929574] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13765.877765] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13767.914639] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13772.218571] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [13774.816031] Lustre: Failing over lustre-MDT0000 [13775.064658] Lustre: server umount lustre-MDT0000 complete [13777.425211] LustreError: 348013:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.203.29@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13777.456490] LustreError: 348013:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 8 previous similar messages [13778.403034] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13778.404160] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13778.420898] Lustre: Skipped 2 previous similar messages [13794.785475] Lustre: 341504:0:(client.c:2490:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1785464455/real 1785464455] req@00000000153e4988 x1872194775916544/t0(0) o400->MGC192.168.203.129@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1785464471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13794.846483] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13796.998809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13805.025278] LustreError: 341502:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000a58708bf x1872194775921920/t0(0) o250->MGC192.168.203.129@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13805.558859] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13805.643514] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [13808.111539] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [13810.668096] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [13810.682260] Lustre: Skipped 2 previous similar messages [13810.766476] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [13810.847559] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13811.357202] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13825.785132] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13828.316906] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [13839.958716] Lustre: server umount lustre-MDT0000 complete [13841.377060] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13841.388838] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13841.419211] Lustre: Skipped 2 previous similar messages [13841.450887] LustreError: 348012:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13841.496581] LustreError: 348012:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 28 previous similar messages [13845.851885] LustreError: 347997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785464522 with bad export cookie 699040579664028321 [13845.857384] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13845.869845] LustreError: 347997:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13846.910197] Lustre: server umount lustre-MDT0001 complete [13853.681113] Lustre: server umount lustre-OST0000 complete [13876.155451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13876.935440] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13883.309442] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13894.590090] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13900.180214] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13910.257437] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13920.068597] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13930.177541] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13939.369485] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [13946.796764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13951.160579] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13960.162454] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13960.172283] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13960.185711] Lustre: Skipped 2 previous similar messages [13960.195702] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13962.213160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13962.225239] Lustre: Skipped 1 previous similar message [13966.058730] Lustre: server umount lustre-OST0000 complete [13972.450493] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13979.050230] Lustre: server umount lustre-MDT0000 complete [13982.707665] LustreError: 355091:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13982.732883] LustreError: 355091:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [13984.689256] LustreError: 355099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785464661 with bad export cookie 699040579664029063 [13984.694000] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13984.722545] LustreError: 355099:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13985.718145] Lustre: server umount lustre-MDT0001 complete [14002.132422] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [14006.287540] Key type lgssc unregistered [14006.634318] LNet: 357913:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14006.643880] LNetError: 357913:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14006.657753] LNet: Removed LNI 192.168.203.129@tcp [14007.666603] Key type .llcrypt unregistered [14007.670557] Key type ._llcrypt unregistered [14054.711683] Key type ._llcrypt registered [14054.717682] Key type .llcrypt registered [14054.880264] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [14056.171679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14056.427708] alg: No test for adler32 (adler32-zlib) [14057.603558] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [14057.814559] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [14059.465041] Key type lgssc registered [14061.437669] Lustre: Echo OBD driver; http://www.lustre.org/ [14072.573833] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14072.599511] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14074.293831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14079.973632] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14083.558805] LustreError: 359438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14083.598754] LustreError: 359438:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14088.692660] LustreError: 359439:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14093.793631] LustreError: 359438:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14095.454369] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14096.518726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14107.209393] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14121.325955] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14131.158731] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14142.548024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14143.231613] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14144.254751] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [14151.365831] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14164.347969] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14181.347700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14181.352771] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14181.380888] Lustre: Skipped 2 previous similar messages [14181.416734] Lustre: Skipped 2 previous similar messages [14186.471102] LustreError: 359435:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14186.495505] LustreError: 359435:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14186.825714] Lustre: server umount lustre-MDT0000 complete [14191.590722] LustreError: 360167:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14191.624601] LustreError: 360167:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [14192.915112] LustreError: 359419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785464869 with bad export cookie 7602757430299105844 [14192.924225] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14192.953748] LustreError: 359419:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14194.101510] Lustre: server umount lustre-MDT0001 complete [14212.498514] Lustre: server umount lustre-OST0000 complete [14234.706177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14235.589128] LustreError: 363182:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14235.716296] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14242.097683] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14255.507861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14255.967920] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14266.312892] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14280.266763] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14293.141346] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14303.434676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14304.440876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14305.476435] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [14316.468575] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14335.570908] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14353.391299] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14353.405633] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14353.428648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14356.961053] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14356.962059] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14356.972685] Lustre: Skipped 1 previous similar message [14357.002048] Lustre: Skipped 1 previous similar message [14359.340847] Lustre: server umount lustre-MDT0000 complete [14362.081844] LustreError: 364618:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14362.107925] LustreError: 364618:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14363.838791] LustreError: 365029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785465040 with bad export cookie 7602757430299106957 [14363.845251] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14363.853796] LustreError: 365029:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14364.400794] Lustre: server umount lustre-MDT0001 complete [14380.350337] Lustre: server umount lustre-OST0000 complete [14400.521646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14401.002573] LustreError: 367102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14401.050716] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14406.968185] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14417.412467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14424.276300] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14435.158479] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14442.442437] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14452.920478] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14453.611563] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14453.622624] Lustre: Skipped 1 previous similar message [14464.755504] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14471.103905] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [14480.838765] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14491.625117] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14491.632429] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14491.651165] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14491.651173] Lustre: Skipped 1 previous similar message [14495.763520] Lustre: server umount lustre-OST0000 complete [14501.858495] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14501.877367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14501.886127] Lustre: Skipped 2 previous similar messages [14501.896144] Lustre: Skipped 1 previous similar message [14508.092094] Lustre: server umount lustre-MDT0000 complete [14512.102479] LustreError: 367102:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14512.132249] LustreError: 367102:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [14513.667224] LustreError: 367083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785465190 with bad export cookie 7602757430299107979 [14513.667834] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14513.712371] LustreError: 367083:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [14514.379392] Lustre: server umount lustre-MDT0001 complete [14537.616733] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [14541.835310] Key type lgssc unregistered [14542.278610] LNet: 370669:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14542.297475] LNetError: 370669:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14542.326699] LNet: Removed LNI 192.168.203.129@tcp [14543.881345] Key type .llcrypt unregistered [14543.886629] Key type ._llcrypt unregistered [14586.029968] Key type ._llcrypt registered [14586.039714] Key type .llcrypt registered [14586.202493] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [14587.651063] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14587.742708] alg: No test for adler32 (adler32-zlib) [14589.465375] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [14589.875933] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [14591.656143] Key type lgssc registered [14594.158258] Lustre: Echo OBD driver; http://www.lustre.org/ [14605.691326] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14605.719167] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14605.739272] LustreError: 372114:0:(osd_handler.c:8554:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [14605.779159] LustreError: 372114:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [14605.795449] LustreError: 372114:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [14605.806024] LustreError: 372114:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [14605.819196] LustreError: 372114:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [14616.195627] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [14616.206860] LustreError: 372513:0:(osd_handler.c:8554:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [14616.255344] LustreError: 372513:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [14616.268802] LustreError: 372513:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [14616.280679] LustreError: 372513:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [14616.292039] LustreError: 372513:0:(super25.c:178:lustre_fill_super()) llite: Unable to mount : rc = -22 [14645.120874] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [14649.125977] Key type lgssc unregistered [14649.668760] LNet: 373536:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14649.678604] LNetError: 373536:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14649.717661] LNet: Removed LNI 192.168.203.129@tcp [14651.516413] Key type .llcrypt unregistered [14651.521919] Key type ._llcrypt unregistered [14680.668484] Key type ._llcrypt registered [14680.671574] Key type .llcrypt registered [14680.877825] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [14716.401473] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [14719.360369] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14719.434716] alg: No test for adler32 (adler32-zlib) [14720.848337] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [14721.319404] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [14723.168640] Key type lgssc registered [14725.038362] Lustre: Echo OBD driver; http://www.lustre.org/ [14786.523894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14786.570852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14788.015747] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14788.119583] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14788.315744] Lustre: lustre-MDT0000: new disk, initializing [14788.435825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14788.471703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14795.534303] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14810.118782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14810.272854] Lustre: 376701:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14810.420595] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14810.425551] Lustre: Skipped 1 previous similar message [14810.548139] Lustre: lustre-MDT0001: new disk, initializing [14810.681150] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14810.744243] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14810.759719] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14816.319570] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14827.917952] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14835.458741] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14845.019689] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14845.520585] Lustre: lustre-OST0000: new disk, initializing [14845.537961] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14845.542332] Lustre: 378672:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14845.692535] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14852.700628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14852.721341] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14852.880019] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14854.934686] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14869.049525] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14873.941513] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14874.163752] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14880.600549] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14881.016228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14888.931648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14888.936034] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14888.954309] LustreError: Skipped 1 previous similar message [14888.958142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14888.997184] Lustre: Skipped 1 previous similar message [14890.613892] Lustre: server umount lustre-OST0000 complete [14897.633276] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14897.636553] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14897.670437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14897.673158] Lustre: Skipped 1 previous similar message [14899.173466] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14899.174907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14901.879642] Lustre: server umount lustre-MDT0000 complete [14906.705802] LustreError: 376694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785465583 with bad export cookie 6260851515225250542 [14906.718781] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14906.723970] LustreError: 376694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [14907.153668] Lustre: server umount lustre-MDT0001 complete [14918.435873] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [14921.045202] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 22:39:54 (1785465594) [14932.478774] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14933.081833] LustreError: 381137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14933.112774] LustreError: 381137:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [14933.220733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14938.596940] LustreError: 381138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14938.650411] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14942.691358] LustreError: 381137:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14947.811820] LustreError: 381138:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14952.325561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14952.830625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14958.922830] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [14968.962517] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14976.832537] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14988.733355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14989.863447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15000.672931] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15016.996507] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15029.101972] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15029.344365] Lustre: lustre-OST0001: new disk, initializing [15029.347675] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [15029.352989] Lustre: 383981:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15029.440042] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [15038.642369] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15039.691168] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [15039.711168] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [15039.826641] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [15052.403991] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15060.035319] Lustre: lustre-OST0001: Not available for connect from 192.168.203.29@tcp (stopping) [15060.051266] Lustre: Skipped 1 previous similar message [15060.480101] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15060.495198] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15060.499626] LustreError: 382963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15060.518854] Lustre: Skipped 1 previous similar message [15060.565421] LustreError: 382963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15060.629185] Lustre: server umount lustre-OST0001 complete [15070.696727] LustreError: 382963:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15070.724494] LustreError: 382963:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15080.934711] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15080.965572] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15080.967928] Lustre: Skipped 1 previous similar message [15080.991970] Lustre: Skipped 1 previous similar message [15091.169882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15091.180686] LustreError: 382964:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15091.182890] Lustre: Skipped 3 previous similar messages [15091.243036] LustreError: 382964:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 7 previous similar messages [15094.240135] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15094.642377] Lustre: server umount lustre-OST0000 complete [15101.410383] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15101.446679] Lustre: Skipped 1 previous similar message [15107.495556] Lustre: server umount lustre-MDT0000 complete [15113.624758] LustreError: 381118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785465790 with bad export cookie 6260851515225251424 [15113.637694] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15113.640520] LustreError: 381118:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15120.810482] Lustre: server umount lustre-MDT0001 complete [15144.394519] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [15148.367705] Key type lgssc unregistered [15148.826917] LNet: 385991:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15148.833476] LNetError: 385991:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15148.854054] LNet: Removed LNI 192.168.203.129@tcp [15149.722925] Key type .llcrypt unregistered [15149.729867] Key type ._llcrypt unregistered [15236.688561] Key type ._llcrypt registered [15236.693584] Key type .llcrypt registered [15236.910139] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [15239.872229] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15240.301982] alg: No test for adler32 (adler32-zlib) [15242.097313] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [15242.825927] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [15244.848231] Key type lgssc registered [15248.968484] Lustre: Echo OBD driver; http://www.lustre.org/ [15265.705763] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15265.724705] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15267.106593] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [15267.120733] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15267.346852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15273.659453] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15284.184095] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15284.418995] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [15284.515232] Lustre: 388802:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15285.247927] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15291.809872] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15303.435061] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15313.636483] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15315.937434] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15315.946601] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15315.977587] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15317.990925] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15317.992221] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15323.106593] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15323.123263] Lustre: Skipped 2 previous similar messages [15328.227419] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15330.272336] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15330.939201] Lustre: server umount lustre-MDT0000 complete [15335.811850] LustreError: 388793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785466012 with bad export cookie 8052376809545323890 [15335.816310] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15335.825541] LustreError: 388793:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15336.329686] Lustre: server umount lustre-MDT0001 complete [15351.953806] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 22:47:06 (1785466026) [15362.183510] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15362.556511] LustreError: 391320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15362.574970] LustreError: 391320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [15362.612170] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15367.655507] LustreError: 391321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15369.271028] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15372.771799] LustreError: 391320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15377.904256] LustreError: 391321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15382.923711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15383.021691] LustreError: 391320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15383.453764] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15388.814106] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15398.974528] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15407.382630] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15418.263639] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15418.718363] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [15419.078134] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15429.777085] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15443.488595] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15454.158812] Lustre: Permanently deactivating lustre-OST0000 [15454.198944] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [15463.022986] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15464.997891] LustreError: 394311:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [15465.010799] LustreError: 394311:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15465.018572] LustreError: 394311:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15467.899869] LustreError: 394359:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15467.909756] LustreError: 394359:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15477.820692] Lustre: server umount lustre-OST0000 complete [15481.405558] LustreError: 394609:0:(mgs_llog.c:2099:mgs_replace_nids()) Only MGS is allowed to be started [15481.414422] LustreError: 394609:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [15485.923102] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15485.932232] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15485.954609] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15485.961886] Lustre: Skipped 1 previous similar message [15487.989659] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15494.638086] Lustre: server umount lustre-MDT0000 complete [15498.210454] LustreError: 392589:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15498.231284] LustreError: 392589:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [15501.125868] LustreError: 392051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785466177 with bad export cookie 8052376809545324471 [15501.128171] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15501.161221] LustreError: 392051:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15510.740568] Lustre: server umount lustre-MDT0001 complete [15522.300098] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15529.688741] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15533.624218] LustreError: 395769:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [15541.409852] Lustre: 396108:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.203.129@tcp [15544.438803] Lustre: server umount MGS complete [15561.368411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15562.110489] LustreError: 396835:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15562.129341] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15562.135228] Lustre: Skipped 1 previous similar message [15562.210240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15568.165882] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15581.140312] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15581.599423] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [15581.602724] LustreError: 388120:0:(client.c:1391:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e9764c41 x1872196711559808/t0(0) o8->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15581.737728] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15589.341727] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15600.156624] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15608.947881] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15612.141606] Lustre: Permanently reactivating lustre-OST0000 [15612.176783] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [15612.185439] Lustre: Skipped 2 previous similar messages [15642.087305] LustreError: 398627:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [15653.814186] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15654.408053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15671.018345] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15685.123316] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15692.257675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15692.266980] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15692.269699] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15692.284880] Lustre: Skipped 1 previous similar message [15692.301709] Lustre: Skipped 5 previous similar messages [15706.080943] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15706.764215] Lustre: server umount lustre-OST0000 complete [15714.785392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15714.796177] LustreError: Skipped 1 previous similar message [15714.800700] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15714.827651] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15714.837104] Lustre: Skipped 4 previous similar messages [15719.539798] Lustre: server umount lustre-MDT0000 complete [15722.990222] LustreError: 398272:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15723.005996] LustreError: 398272:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [15724.085704] LustreError: 396817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785466400 with bad export cookie 8052376809545325521 [15724.091560] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15724.105710] LustreError: 396817:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15724.719741] Lustre: server umount lustre-MDT0001 complete [15742.410918] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [15746.385987] Key type lgssc unregistered [15746.900394] LNet: 401176:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15746.906413] LNetError: 401176:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15747.950614] LNet: Removed LNI 192.168.203.129@tcp [15749.064432] Key type .llcrypt unregistered [15749.072225] Key type ._llcrypt unregistered [15778.738765] Key type ._llcrypt registered [15778.748095] Key type .llcrypt registered [15778.919452] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [15799.084657] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [15801.253612] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15801.344712] alg: No test for adler32 (adler32-zlib) [15802.596701] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [15803.020655] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [15804.848174] Key type lgssc registered [15806.783193] Lustre: Echo OBD driver; http://www.lustre.org/ [15861.258061] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15861.314661] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15862.830506] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15862.881351] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15862.957776] Lustre: lustre-MDT0000: new disk, initializing [15863.045115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15863.080349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15868.738176] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15882.300888] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15882.527431] Lustre: 404340:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15882.641141] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15882.649460] Lustre: Skipped 1 previous similar message [15882.780243] Lustre: lustre-MDT0001: new disk, initializing [15882.963068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15883.032702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15883.050374] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15889.337449] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15902.175292] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15909.207637] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15918.174599] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15918.645229] Lustre: lustre-OST0000: new disk, initializing [15918.660231] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15918.675574] Lustre: 406310:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15918.873679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15926.886051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15926.913849] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15927.122046] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15927.945979] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [15941.879629] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15948.576361] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [15949.033639] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [15955.780414] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [15956.055644] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [15963.107311] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15963.108094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15963.129757] Lustre: Skipped 1 previous similar message [15963.158434] Lustre: Skipped 1 previous similar message [15965.122515] Lustre: server umount lustre-OST0000 complete [15970.274662] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15970.283087] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15970.300297] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15973.346594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15973.348307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15973.386434] Lustre: Skipped 1 previous similar message [15976.856925] Lustre: server umount lustre-MDT0000 complete [15982.141943] LustreError: 406309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785466658 with bad export cookie 137793121389951277 [15982.150090] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15982.152546] LustreError: 406309:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15982.744680] Lustre: server umount lustre-MDT0001 complete [15995.315824] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [15997.122913] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 22:57:51 (1785466671) [16005.890651] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16006.608551] LustreError: 408771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16006.627131] LustreError: 408771:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16006.722219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16011.753078] LustreError: 408772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16012.052772] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16016.875890] LustreError: 408771:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16021.992452] LustreError: 408772:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16023.983743] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16024.547514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16030.979900] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16041.181562] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16050.989342] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16061.896896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16062.528593] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16071.514405] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16082.355395] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16085.144159] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [16088.280066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [16097.249613] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16097.272288] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16097.275458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16097.293992] Lustre: Skipped 1 previous similar message [16102.376025] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16102.384558] Lustre: Skipped 2 previous similar messages [16102.901718] Lustre: server umount lustre-OST0000 complete [16111.603788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16111.621122] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16111.656853] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16114.156574] Lustre: server umount lustre-MDT0000 complete [16117.733964] LustreError: 409501:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16117.764542] LustreError: 409501:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16119.196257] LustreError: 408752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785466795 with bad export cookie 137793121389952159 [16119.198681] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16119.209894] LustreError: 408752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [16119.712553] Lustre: server umount lustre-MDT0001 complete [16135.066785] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [16139.293466] Key type lgssc unregistered [16139.614433] LNet: 412502:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16139.624469] LNetError: 412502:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16139.661559] LNet: Removed LNI 192.168.203.129@tcp [16140.760616] Key type .llcrypt unregistered [16140.764758] Key type ._llcrypt unregistered [16173.928394] Key type ._llcrypt registered [16173.933964] Key type .llcrypt registered [16174.081742] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [16174.508364] Key type .llcrypt unregistered [16174.513219] Key type ._llcrypt unregistered [16203.892105] Key type ._llcrypt registered [16203.893691] Key type .llcrypt registered [16204.150456] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [16206.179208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16206.253133] alg: No test for adler32 (adler32-zlib) [16207.702886] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [16208.225551] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [16210.008331] Key type lgssc registered [16212.446725] Lustre: Echo OBD driver; http://www.lustre.org/ [16223.512110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16223.548299] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16225.316715] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16232.243714] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16234.468344] LustreError: 414731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16234.492723] LustreError: 414731:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16239.597435] LustreError: 414732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16244.178943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16244.674381] LustreError: 414731:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16244.695387] LustreError: 414731:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16244.702394] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16249.826147] LustreError: 414732:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16249.846414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [16249.855336] LustreError: 414732:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16253.479573] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16254.954176] LustreError: 414726:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16254.989293] LustreError: 414726:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16263.283892] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16265.187161] LustreError: 414728:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16265.228263] LustreError: 414728:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16271.823295] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16272.466857] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16278.367191] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16292.834073] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16292.847179] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16292.860432] Lustre: Skipped 1 previous similar message [16292.879048] Lustre: Skipped 1 previous similar message [16297.523437] Lustre: server umount lustre-OST0000 complete [16303.075775] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16303.079909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16303.094938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16308.194114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16308.209089] Lustre: Skipped 2 previous similar messages [16309.647533] Lustre: server umount lustre-MDT0000 complete [16313.321794] LustreError: 416667:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16313.352703] LustreError: 416667:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16315.394551] LustreError: 414711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785466992 with bad export cookie 15243062169558237212 [16315.404639] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16315.420239] LustreError: 414711:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16316.855960] Lustre: server umount lustre-MDT0001 complete [16334.167022] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [16338.969558] Key type lgssc unregistered [16339.473079] LNet: 417885:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16339.492307] LNetError: 417885:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16340.520618] LNet: Removed LNI 192.168.203.129@tcp [16341.749545] Key type .llcrypt unregistered [16341.752224] Key type ._llcrypt unregistered [16380.646323] Key type ._llcrypt registered [16380.647819] Key type .llcrypt registered [16380.892735] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [16382.988054] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16383.426837] alg: No test for adler32 (adler32-zlib) [16384.850794] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [16385.218098] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [16387.104616] Key type lgssc registered [16389.433470] Lustre: Echo OBD driver; http://www.lustre.org/ [16404.272657] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16404.291929] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16419.808478] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [16424.928972] LustreError: 419280:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.203.129@tcp: failed processing log, type 4: rc = -110 [16451.680376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16463.123945] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16475.866489] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16486.052667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16486.944797] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16487.051156] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [16492.835402] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16493.031686] LustreError: 420383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16493.057753] LustreError: 420383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [16498.169015] LustreError: 420383:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16498.198286] LustreError: 420383:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16503.283135] LustreError: 420320:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16503.301630] LustreError: 420320:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16503.767230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16504.339347] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16511.425191] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16520.717492] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16528.549493] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16538.089553] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16538.099342] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16538.117744] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16540.130725] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16540.142173] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16540.151881] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16543.736538] Lustre: server umount lustre-OST0000 complete [16550.369554] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16550.374943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16550.400178] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16550.447431] Lustre: Skipped 1 previous similar message [16550.477348] Lustre: Skipped 2 previous similar messages [16551.022478] Lustre: server umount lustre-MDT0000 complete [16555.493925] LustreError: 420323:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16555.518206] LustreError: 420323:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16555.734034] LustreError: 419288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785467232 with bad export cookie 8115990846374829156 [16555.744889] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16555.749507] LustreError: 419288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16556.587722] Lustre: server umount lustre-MDT0001 complete [16578.914461] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [16585.156468] Key type lgssc unregistered [16585.820310] LNet: 422874:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16585.835889] LNetError: 422874:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16585.895630] LNet: Removed LNI 192.168.203.129@tcp [16588.318172] Key type .llcrypt unregistered [16588.326754] Key type ._llcrypt unregistered [16626.640157] Key type ._llcrypt registered [16626.644891] Key type .llcrypt registered [16626.836074] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [16628.039173] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16628.147698] alg: No test for adler32 (adler32-zlib) [16629.501788] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [16629.846984] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [16631.489385] Key type lgssc registered [16632.776390] Lustre: Echo OBD driver; http://www.lustre.org/ [16642.426806] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16642.474823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16643.901292] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16650.922530] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16654.309367] LustreError: 424294:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16654.329263] LustreError: 424294:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16659.449386] LustreError: 424295:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16661.501775] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16662.047915] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16668.429356] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16678.649259] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16687.301795] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16697.953247] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16698.604393] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16707.606045] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16714.134059] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [16717.764427] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16723.426362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16723.433853] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16723.465357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16724.449851] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16724.451855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16724.464157] Lustre: Skipped 1 previous similar message [16724.493228] Lustre: Skipped 1 previous similar message [16725.290939] Lustre: server umount lustre-MDT0000 complete [16729.572589] LustreError: 424289:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16729.593322] LustreError: 424289:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16729.764700] LustreError: 424275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.203.29@tcp arrived at 1785467406 with bad export cookie 5382709665602986117 [16732.934337] LustreError: 424274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785467409 with bad export cookie 5382709665602985221 [16732.958761] LustreError: 424274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16732.988013] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16734.689260] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16734.690465] LustreError: 424290:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16734.727609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16734.752800] LustreError: 424290:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16734.785499] Lustre: Skipped 2 previous similar messages [16737.697723] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16739.533203] Lustre: server umount lustre-OST0000 complete [16742.823139] LustreError: 425023:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16742.859637] LustreError: 425023:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [16751.660556] Lustre: server umount lustre-MDT0001 complete [16769.533702] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [16774.105328] Key type lgssc unregistered [16774.620552] LNet: 428002:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16774.632967] LNetError: 428002:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16774.685727] LNet: Removed LNI 192.168.203.129@tcp [16776.280140] Key type .llcrypt unregistered [16776.287226] Key type ._llcrypt unregistered [16810.397297] Key type ._llcrypt registered [16810.403160] Key type .llcrypt registered [16810.568658] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [16812.261139] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16812.665302] alg: No test for adler32 (adler32-zlib) [16814.086729] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [16814.482371] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [16816.312250] Key type lgssc registered [16817.674063] Lustre: Echo OBD driver; http://www.lustre.org/ [16827.591330] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16827.626305] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16829.439080] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16835.639210] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16839.674527] LustreError: 429420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16839.704339] LustreError: 429420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16844.786181] LustreError: 429421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16849.035325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16849.556970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16855.076815] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16866.569547] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16874.656573] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16884.149630] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16884.728783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16892.634762] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [16900.089948] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [16902.240093] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16910.316506] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16910.438665] LustreError: 429421:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16910.451931] LustreError: 429421:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16910.786388] Lustre: server umount lustre-MDT0001 complete [16915.428547] LustreError: 429417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16915.462903] LustreError: 429417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [16920.546635] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16920.548197] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16920.548660] LustreError: 429420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16920.557653] Lustre: Skipped 1 previous similar message [16920.559177] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16925.679135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16926.711859] Lustre: server umount lustre-OST0000 complete [16930.786935] LustreError: 430374:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16930.806842] LustreError: 430374:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [16931.899899] Lustre: server umount lustre-MDT0000 complete [16951.819509] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [16956.487377] Key type lgssc unregistered [16957.058816] LNet: 433110:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16957.066906] LNetError: 433110:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16958.120616] LNet: Removed LNI 192.168.203.129@tcp [16959.630262] Key type .llcrypt unregistered [16959.633895] Key type ._llcrypt unregistered [16994.831986] Key type ._llcrypt registered [16994.833761] Key type .llcrypt registered [16994.970223] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [17019.818355] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [17022.068340] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17022.127560] alg: No test for adler32 (adler32-zlib) [17023.264387] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [17023.545444] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [17025.336107] Key type lgssc registered [17026.801442] Lustre: Echo OBD driver; http://www.lustre.org/ [17079.175522] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [17098.834750] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17098.858303] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17100.186426] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17100.206190] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17100.354745] Lustre: lustre-MDT0000: new disk, initializing [17100.456572] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17100.472736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17105.765208] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [17125.318026] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17125.536836] Lustre: lustre-OST0000: new disk, initializing [17125.540810] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17125.544311] Lustre: Skipped 1 previous similar message [17125.549641] Lustre: 438172:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17125.607531] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17133.143604] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [17133.714154] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [17133.732221] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [17133.784269] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [17146.084119] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17164.749165] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17164.912314] Lustre: 437154:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17165.034083] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17165.137239] Lustre: lustre-MDT0001: new disk, initializing [17165.285199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17165.344049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [17165.353105] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [17171.160583] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [17183.464783] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [17183.754957] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [17188.736170] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [17188.992864] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [17199.587555] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [17199.595157] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17199.615868] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17199.636089] Lustre: Skipped 1 previous similar message [17199.651568] Lustre: Skipped 2 previous similar messages [17204.713367] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17209.830762] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [17209.860039] Lustre: Skipped 3 previous similar messages [17210.338679] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17210.963114] Lustre: server umount lustre-MDT0001 complete [17220.071100] LustreError: 437165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17220.101850] LustreError: 437165:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 5 previous similar messages [17225.199807] LustreError: 437159:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17225.220631] LustreError: 437159:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17230.305900] LustreError: 437164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17230.344582] LustreError: 437164:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17235.428894] LustreError: 437165:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17235.444716] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17235.465895] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17235.479155] Lustre: Skipped 1 previous similar message [17239.614776] Lustre: server umount lustre-OST0000 complete [17240.547646] LustreError: 437164:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17251.856473] Lustre: server umount lustre-MDT0000 complete [17269.691687] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [17273.311883] Key type lgssc unregistered [17273.698187] LNet: 441925:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17273.714815] LNetError: 441925:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17273.739709] LNet: Removed LNI 192.168.203.129@tcp [17275.163073] Key type .llcrypt unregistered [17275.164934] Key type ._llcrypt unregistered [17330.220844] Key type ._llcrypt registered [17330.222787] Key type .llcrypt registered [17330.366089] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [17330.647092] Key type .llcrypt unregistered [17330.652093] Key type ._llcrypt unregistered [17351.971303] Key type ._llcrypt registered [17351.984241] Key type .llcrypt registered [17352.156771] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [17353.756601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [17354.049752] alg: No test for adler32 (adler32-zlib) [17355.305448] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [17355.666912] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [17357.388564] Key type lgssc registered [17359.593712] Lustre: Echo OBD driver; http://www.lustre.org/ [17403.536283] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17403.568584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17404.977883] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17405.042920] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17405.168683] Lustre: lustre-MDT0000: new disk, initializing [17405.321461] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17405.357332] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17413.623801] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [17429.686927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17429.946953] Lustre: 445308:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17430.095758] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17430.108977] Lustre: Skipped 1 previous similar message [17430.311406] Lustre: lustre-MDT0001: new disk, initializing [17430.480514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17430.575993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17430.580433] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17436.210538] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [17448.612136] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17458.398660] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17469.090425] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17469.521503] Lustre: lustre-OST0000: new disk, initializing [17469.531419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17469.539394] Lustre: 447280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17469.618322] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17475.200214] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17475.221834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17475.404344] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17479.429575] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [17496.222477] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17511.394673] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17511.405717] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17511.422083] Lustre: Skipped 1 previous similar message [17511.442934] Lustre: Skipped 2 previous similar messages [17518.163639] Lustre: server umount lustre-MDT0000 complete [17521.642870] LustreError: 445319:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17521.673967] LustreError: 445319:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 6 previous similar messages [17523.435644] LustreError: 447279:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785468200 with bad export cookie 3944560693939395719 [17523.450821] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17523.907480] Lustre: server umount lustre-MDT0001 complete [17539.407492] Lustre: server umount lustre-OST0000 complete [17553.560695] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 23:23:47 (1785468227) [17563.599368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17564.224444] LustreError: 449325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17564.350347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17569.767911] LustreError: 449326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17570.888167] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [17573.859918] LustreError: 449325:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17578.984058] LustreError: 449326:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17582.301037] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17582.833836] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17588.637708] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [17598.565582] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17606.702411] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17628.605990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17628.743055] Lustre: Found index 0 for lustre-OST0000, updating log [17628.997764] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17634.101255] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [17637.447250] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [17648.171327] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17654.756340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [17654.759335] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17654.787250] Lustre: Skipped 1 previous similar message [17654.807020] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17655.850911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17655.859682] Lustre: Skipped 1 previous similar message [17657.896026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17657.904796] Lustre: Skipped 1 previous similar message [17660.561038] Lustre: server umount lustre-OST0000 complete [17677.229791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17677.253120] Lustre: Skipped 1 previous similar message [17677.256949] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17677.260663] Lustre: Skipped 2 previous similar messages [17683.946445] Lustre: server umount lustre-MDT0000 complete [17685.491414] LustreError: 449321:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [17685.520644] LustreError: 449321:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [17688.855235] LustreError: 449305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785468365 with bad export cookie 3944560693939397112 [17688.862908] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17688.872712] LustreError: 449305:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [17689.335367] Lustre: server umount lustre-MDT0001 complete [17695.779314] LNet: 1 local NIs in recovery (showing 1): 192.168.203.129@tcp [17699.916708] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 23:26:14 (1785468374) [18343.342835] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18344.068813] LustreError: 455742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18344.093447] LustreError: 455742:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [18344.203381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18349.561371] LustreError: 455743:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18349.809771] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [18353.636538] LustreError: 455742:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18362.485625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18362.927172] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18370.520054] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [18387.905288] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18399.995183] LNet: 1 local NIs in recovery (showing 1): 192.168.203.129@tcp [18400.504609] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18426.366440] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18426.688640] Lustre: Found index 0 for lustre-OST0000, updating log [18427.236651] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18435.351033] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [18445.652575] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18448.770826] Lustre: server umount lustre-OST0000 complete [18454.755475] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18454.762672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18454.793813] Lustre: Skipped 2 previous similar messages [18454.822284] Lustre: Skipped 4 previous similar messages [18455.841087] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18457.891689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18457.903138] Lustre: Skipped 4 previous similar messages [18461.985913] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18461.992149] Lustre: Skipped 7 previous similar messages [18463.648164] LNet: 1 local NIs in recovery (showing 1): 192.168.203.129@tcp [18468.320454] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [18468.723855] Lustre: server umount lustre-MDT0000 complete [18469.159976] LustreError: 456642:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [18469.194764] LustreError: 456642:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [18473.842757] LustreError: 455723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785469150 with bad export cookie 3944560693939398190 [18473.865290] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18474.652436] Lustre: server umount lustre-MDT0001 complete [18513.136690] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [18526.046290] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [18578.426993] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 23:40:51 (1785469251) [18603.130148] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [18607.784858] Key type lgssc unregistered [18608.227754] LNet: 462358:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18608.232892] LNetError: 462358:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18736.235194] LNet: Removed LNI 192.168.203.129@tcp [18737.929247] Key type .llcrypt unregistered [18737.931368] Key type ._llcrypt unregistered [18762.363322] Key type ._llcrypt registered [18762.365709] Key type .llcrypt registered [18762.612469] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [18764.245800] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [18764.375821] alg: No test for adler32 (adler32-zlib) [18766.004612] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [18766.529347] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [18768.305185] Key type lgssc registered [18770.669287] Lustre: Echo OBD driver; http://www.lustre.org/ [18776.149324] LNet: Added LNI 192.168.203.129@tcp100 [8/256/0/180] [18777.425755] LNet: Added LNI 192.168.203.129@tcp101 [8/256/0/180] [18778.538607] LNet: Added LNI 192.168.203.129@tcp102 [8/256/0/180] [18779.578862] LNet: Added LNI 192.168.203.129@tcp103 [8/256/0/180] [18780.750611] LNet: Added LNI 192.168.203.129@tcp104 [8/256/0/180] [18781.932349] LNet: Added LNI 192.168.203.129@tcp105 [8/256/0/180] [18783.044787] LNet: Added LNI 192.168.203.129@tcp106 [8/256/0/180] [18784.149551] LNet: Added LNI 192.168.203.129@tcp107 [8/256/0/180] [18785.220253] LNet: Added LNI 192.168.203.129@tcp108 [8/256/0/180] [18786.321414] LNet: Added LNI 192.168.203.129@tcp109 [8/256/0/180] [18787.480396] LNet: Added LNI 192.168.203.129@tcp110 [8/256/0/180] [18788.509253] LNet: Added LNI 192.168.203.129@tcp111 [8/256/0/180] [18789.683707] LNet: Added LNI 192.168.203.129@tcp112 [8/256/0/180] [18790.834899] LNet: Added LNI 192.168.203.129@tcp113 [8/256/0/180] [18792.016765] LNet: Added LNI 192.168.203.129@tcp114 [8/256/0/180] [18793.159711] LNet: Added LNI 192.168.203.129@tcp115 [8/256/0/180] [18794.108220] LNet: Added LNI 192.168.203.129@tcp116 [8/256/0/180] [18795.090450] LNet: Added LNI 192.168.203.129@tcp117 [8/256/0/180] [18796.203111] LNet: Added LNI 192.168.203.129@tcp118 [8/256/0/180] [18797.428676] LNet: Added LNI 192.168.203.129@tcp119 [8/256/0/180] [18798.616363] LNet: Added LNI 192.168.203.129@tcp120 [8/256/0/180] [18799.775538] LNet: Added LNI 192.168.203.129@tcp121 [8/256/0/180] [18800.814685] LNet: Added LNI 192.168.203.129@tcp122 [8/256/0/180] [18801.819365] LNet: Added LNI 192.168.203.129@tcp123 [8/256/0/180] [18802.746938] LNet: Added LNI 192.168.203.129@tcp124 [8/256/0/180] [18803.936552] LNet: Added LNI 192.168.203.129@tcp125 [8/256/0/180] [18805.076932] LNet: Added LNI 192.168.203.129@tcp126 [8/256/0/180] [18806.265506] LNet: Added LNI 192.168.203.129@tcp127 [8/256/0/180] [18807.474724] LNet: Added LNI 192.168.203.129@tcp128 [8/256/0/180] [18808.854152] LNet: Added LNI 192.168.203.129@tcp129 [8/256/0/180] [18810.167506] LNet: Added LNI 192.168.203.129@tcp130 [8/256/0/180] [18811.290558] LNet: Added LNI 192.168.203.129@tcp131 [8/256/0/180] [18812.176256] LNet: Added LNI 192.168.203.129@tcp132 [8/256/0/180] [18813.238952] LNet: Added LNI 192.168.203.129@tcp133 [8/256/0/180] [18814.323362] LNet: Added LNI 192.168.203.129@tcp134 [8/256/0/180] [18815.006147] LNet: Added LNI 192.168.203.129@tcp135 [8/256/0/180] [18823.816113] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [18823.863317] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18825.046609] Lustre: 465402:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.129@tcp: can't add NID 192.168.203.129@tcp130: rc = -75 [18825.242088] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [18825.288477] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [18825.382448] Lustre: lustre-MDT0000: new disk, initializing [18825.554427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [18825.594593] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [18831.047583] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [18847.020220] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [18847.312105] Lustre: 465420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [18847.461958] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [18847.467407] Lustre: Skipped 1 previous similar message [18847.620834] Lustre: lustre-MDT0001: new disk, initializing [18847.683592] Lustre: 466217:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.129@tcp: can't add NID 192.168.203.129@tcp130: rc = -75 [18847.693460] Lustre: 466217:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18847.868550] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [18847.925009] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [18847.947362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [18854.565472] Lustre: 466436:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.129@tcp: can't add NID 192.168.203.129@tcp130: rc = -75 [18854.582235] Lustre: 466436:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18855.318096] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [18869.349209] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [18878.354370] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [18889.577972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [18889.974995] Lustre: lustre-OST0000: new disk, initializing [18889.981596] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [18889.991120] Lustre: 467391:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [18890.102169] Lustre: 467396:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.129@tcp: can't add NID 192.168.203.129@tcp130: rc = -75 [18890.108788] Lustre: 467396:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [18890.150454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [18896.384498] Lustre: 467676:0:(lustre_peer.c:124:class_add_uuid()) 192.168.203.129@tcp: can't add NID 192.168.203.129@tcp130: rc = -75 [18896.393364] Lustre: 467676:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [18896.473655] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [18896.496582] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [18896.657549] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [18898.053346] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [18909.192051] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [18922.466870] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18922.520500] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [18922.526316] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [18922.541217] Lustre: Skipped 1 previous similar message [18924.836114] Lustre: server umount lustre-OST0000 complete [18932.706655] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [18932.710197] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [18932.719924] Lustre: Skipped 2 previous similar messages [18932.743122] Lustre: Skipped 1 previous similar message [18936.731719] Lustre: server umount lustre-MDT0000 complete [18941.294991] LustreError: 465411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785469617 with bad export cookie 8951922532119056104 [18941.304280] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [18941.312877] LustreError: 465411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [18941.795576] Lustre: server umount lustre-MDT0001 complete [18957.043848] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [18961.269313] Key type lgssc unregistered [18961.994237] LNet: 469306:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [18962.010610] LNetError: 469306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18962.018459] LNet: Removed LNI 192.168.203.129@tcp [18963.040243] LNet: Removed LNI 192.168.203.129@tcp100 [18963.049220] LNetError: 469306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18963.056808] LNetError: 469306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18964.066945] LNet: Removed LNI 192.168.203.129@tcp101 [18964.073960] LNetError: 469306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18965.088643] LNet: Removed LNI 192.168.203.129@tcp102 [18966.176245] LNet: Removed LNI 192.168.203.129@tcp103 [18966.178133] LNetError: 469306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [18966.195151] LNetError: 469306:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [18966.213084] LNet: Removed LNI 192.168.203.129@tcp104 [18967.264488] LNet: Removed LNI 192.168.203.129@tcp105 [18968.288223] LNet: Removed LNI 192.168.203.129@tcp106 [18968.290276] LNet: Removed LNI 192.168.203.129@tcp107 [18968.298938] LNet: Removed LNI 192.168.203.129@tcp108 [18968.311932] LNet: Removed LNI 192.168.203.129@tcp109 [18968.317634] LNet: Removed LNI 192.168.203.129@tcp110 [18968.323930] LNet: Removed LNI 192.168.203.129@tcp111 [18968.333418] LNet: Removed LNI 192.168.203.129@tcp112 [18968.339080] LNet: Removed LNI 192.168.203.129@tcp113 [18968.345919] LNet: Removed LNI 192.168.203.129@tcp114 [18968.354500] LNet: Removed LNI 192.168.203.129@tcp115 [18968.366414] LNet: Removed LNI 192.168.203.129@tcp116 [18968.371444] LNet: Removed LNI 192.168.203.129@tcp117 [18968.385481] LNet: Removed LNI 192.168.203.129@tcp118 [18968.393103] LNet: Removed LNI 192.168.203.129@tcp119 [18968.404493] LNet: Removed LNI 192.168.203.129@tcp120 [18968.411584] LNet: Removed LNI 192.168.203.129@tcp121 [18968.419147] LNet: Removed LNI 192.168.203.129@tcp122 [18968.428473] LNet: Removed LNI 192.168.203.129@tcp123 [18968.434354] LNet: Removed LNI 192.168.203.129@tcp124 [18968.457007] LNet: Removed LNI 192.168.203.129@tcp125 [18968.463392] LNet: Removed LNI 192.168.203.129@tcp126 [18968.470792] LNet: Removed LNI 192.168.203.129@tcp127 [18968.486920] LNet: Removed LNI 192.168.203.129@tcp128 [18968.492630] LNet: Removed LNI 192.168.203.129@tcp129 [18968.500325] LNet: Removed LNI 192.168.203.129@tcp130 [18968.507210] LNet: Removed LNI 192.168.203.129@tcp131 [18968.521402] LNet: Removed LNI 192.168.203.129@tcp132 [18968.532215] LNet: Removed LNI 192.168.203.129@tcp133 [18968.540150] LNet: Removed LNI 192.168.203.129@tcp134 [18968.567496] LNet: Removed LNI 192.168.203.129@tcp135 [18970.048135] Key type .llcrypt unregistered [18970.055339] Key type ._llcrypt unregistered [19002.109314] Key type ._llcrypt registered [19002.121135] Key type .llcrypt registered [19002.262798] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [19025.128439] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [19026.401192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19026.484181] alg: No test for adler32 (adler32-zlib) [19028.134837] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [19028.570142] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [19030.384202] Key type lgssc registered [19032.603103] Lustre: Echo OBD driver; http://www.lustre.org/ [19081.605668] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 23:49:15 (1785469755) [19107.101486] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [19111.037975] Key type lgssc unregistered [19111.393596] LNet: 473400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19111.416074] LNetError: 473400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19111.443513] LNet: Removed LNI 192.168.203.129@tcp [19113.790530] Key type .llcrypt unregistered [19113.795042] Key type ._llcrypt unregistered [19144.556820] Key type ._llcrypt registered [19144.558536] Key type .llcrypt registered [19144.728062] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [19146.061148] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19146.245612] alg: No test for adler32 (adler32-zlib) [19148.080110] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [19148.613289] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [19150.448197] Key type lgssc registered [19151.926714] Lustre: Echo OBD driver; http://www.lustre.org/ [19166.236371] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [19166.272763] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19167.784255] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [19167.823589] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [19167.928483] Lustre: lustre-MDT0000: new disk, initializing [19168.049751] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19168.067609] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [19174.958193] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [19190.481177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19190.648275] Lustre: 474613:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [19190.768721] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [19190.775958] Lustre: Skipped 1 previous similar message [19190.886924] Lustre: lustre-MDT0001: new disk, initializing [19191.064775] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19191.120512] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [19191.151854] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [19196.718925] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [19209.331047] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19218.193679] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19228.892601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [19229.406214] Lustre: lustre-OST0000: new disk, initializing [19229.427429] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [19229.444829] Lustre: 476584:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [19229.626091] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [19238.540091] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [19238.548496] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [19238.734967] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [19240.429870] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [19258.864472] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [19265.312923] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing lnet_if_list [19274.952331] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [19274.972609] LNet: Added LNI 192.168.203.129@tcp10 [8/256/0/180] [19285.477978] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [19285.548834] LNet: Added LNI 192.168.203.129@tcp11 [8/256/0/180] [19292.382460] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [19292.418989] LNet: Added LNI 192.168.203.129@tcp12 [8/256/0/180] [19300.589678] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [19300.648371] LNet: Added LNI 192.168.203.129@tcp13 [8/256/0/180] [19310.474493] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [19310.527835] LNet: Added LNI 192.168.203.129@tcp14 [8/256/0/180] [19318.923679] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [19318.971962] LNet: Added LNI 192.168.203.129@tcp15 [8/256/0/180] [19326.076316] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [19326.133434] LNet: Added LNI 192.168.203.129@tcp16 [8/256/0/180] [19333.554961] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [19333.604142] LNet: Added LNI 192.168.203.129@tcp17 [8/256/0/180] [19339.991772] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [19340.052062] LNet: Added LNI 192.168.203.129@tcp18 [8/256/0/180] [19347.221309] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [19347.243829] LNet: Added LNI 192.168.203.129@tcp19 [8/256/0/180] [19353.623658] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [19353.720277] LNet: Added LNI 192.168.203.129@tcp20 [8/256/0/180] [19361.080263] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [19361.152823] LNet: Added LNI 192.168.203.129@tcp21 [8/256/0/180] [19368.464536] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [19368.510847] LNet: Added LNI 192.168.203.129@tcp22 [8/256/0/180] [19376.022367] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [19376.057601] LNet: Added LNI 192.168.203.129@tcp23 [8/256/0/180] [19383.341471] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [19383.405638] LNet: Added LNI 192.168.203.129@tcp24 [8/256/0/180] [19391.148277] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [19391.192871] LNet: Added LNI 192.168.203.129@tcp25 [8/256/0/180] [19400.914287] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [19400.989796] LNet: Added LNI 192.168.203.129@tcp26 [8/256/0/180] [19408.152478] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [19408.215030] LNet: Added LNI 192.168.203.129@tcp27 [8/256/0/180] [19417.918099] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [19417.969413] LNet: Added LNI 192.168.203.129@tcp28 [8/256/0/180] [19425.567035] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [19425.607736] LNet: Added LNI 192.168.203.129@tcp29 [8/256/0/180] [19433.861327] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [19433.907642] LNet: Added LNI 192.168.203.129@tcp30 [8/256/0/180] [19443.365505] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [19443.400555] LNet: Added LNI 192.168.203.129@tcp31 [8/256/0/180] [19451.796248] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [19451.843304] LNet: Added LNI 192.168.203.129@tcp32 [8/256/0/180] [19461.127873] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [19461.199926] LNet: Added LNI 192.168.203.129@tcp33 [8/256/0/180] [19470.312591] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [19470.368340] LNet: Added LNI 192.168.203.129@tcp34 [8/256/0/180] [19482.875027] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [19482.945185] LNet: Added LNI 192.168.203.129@tcp35 [8/256/0/180] [19498.116708] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [19498.176242] LNet: Added LNI 192.168.203.129@tcp36 [8/256/0/180] [19510.558175] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [19510.630947] LNet: Added LNI 192.168.203.129@tcp37 [8/256/0/180] [19522.285172] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [19522.333156] LNet: Added LNI 192.168.203.129@tcp38 [8/256/0/180] [19532.159717] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [19532.190741] LNet: Added LNI 192.168.203.129@tcp39 [8/256/0/180] [19542.634207] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [19542.732378] LNet: Added LNI 192.168.203.129@tcp40 [8/256/0/180] [19555.177154] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [19555.245176] LNet: Added LNI 192.168.203.129@tcp41 [8/256/0/180] [19567.704636] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [19567.726640] LNet: Added LNI 192.168.203.129@tcp42 [8/256/0/180] [19580.759301] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [19580.814974] LNet: Added LNI 192.168.203.129@tcp43 [8/256/0/180] [19594.290331] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [19594.332589] LNet: Added LNI 192.168.203.129@tcp44 [8/256/0/180] [19606.182409] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [19606.283055] LNet: Added LNI 192.168.203.129@tcp45 [8/256/0/180] [19620.599537] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [19620.656744] LNet: Added LNI 192.168.203.129@tcp46 [8/256/0/180] [19633.467951] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [19633.535193] LNet: Added LNI 192.168.203.129@tcp47 [8/256/0/180] [19644.672945] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [19644.714164] LNet: Added LNI 192.168.203.129@tcp48 [8/256/0/180] [19660.344036] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [19660.430685] LNet: Added LNI 192.168.203.129@tcp49 [8/256/0/180] [19678.067928] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [19678.105451] LNet: Added LNI 192.168.203.129@tcp50 [8/256/0/180] [19703.268221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19703.277453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19703.296832] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19704.817677] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19704.818020] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19704.838533] Lustre: Skipped 1 previous similar message [19704.851373] Lustre: Skipped 1 previous similar message [19708.800235] Lustre: server umount lustre-MDT0000 complete [19714.326411] LustreError: 486807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785470390 with bad export cookie 11916932469638691250 [19714.331692] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19714.342927] LustreError: 486807:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [19715.046215] LustreError: 475420:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19715.051876] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19715.061071] LustreError: 475420:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [19715.349207] Lustre: server umount lustre-MDT0001 complete [19740.882612] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [19753.878731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19759.073340] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xa56172f1492204e1 [19759.099173] Lustre: MGC192.168.203.129@tcp: Connection restored to 0@lo (at 0@lo) [19759.647129] LustreError: 476578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19759.677928] LustreError: 476578:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 1 previous similar message [19759.733620] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [19765.222158] LustreError: 476579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19766.260844] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [19769.351864] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19769.351957] LustreError: 476578:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19774.450224] LustreError: 476579:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19779.879343] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [19780.647646] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [19785.710131] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [19788.396923] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [19799.302940] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [19806.024839] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [19836.385443] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [19836.398881] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19836.435221] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [19836.438035] Lustre: Skipped 1 previous similar message [19838.482635] Lustre: server umount lustre-OST0000 complete [19847.140031] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [19847.143602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [19847.151578] LustreError: Skipped 1 previous similar message [19847.154341] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [19847.154347] Lustre: Skipped 1 previous similar message [19847.159188] Lustre: Skipped 2 previous similar messages [19850.520222] Lustre: server umount lustre-MDT0000 complete [19852.276268] LustreError: 500863:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [19852.311523] LustreError: 500863:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 3 previous similar messages [19856.810126] LustreError: 474606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785470533 with bad export cookie 11916932469638694113 [19856.814784] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [19856.840768] LustreError: 474606:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [19857.496439] Lustre: server umount lustre-MDT0001 complete [19871.705893] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [19875.550939] Key type lgssc unregistered [19876.165963] LNet: 503683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [19876.175282] LNetError: 503683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [19876.188048] LNet: Removed LNI 192.168.203.129@tcp [19876.195156] LNet: Removed LNI 192.168.203.129@tcp10 [19876.208879] LNet: Removed LNI 192.168.203.129@tcp11 [19876.214329] LNet: Removed LNI 192.168.203.129@tcp12 [19876.224106] LNet: Removed LNI 192.168.203.129@tcp13 [19876.229685] LNet: Removed LNI 192.168.203.129@tcp14 [19876.235745] LNet: Removed LNI 192.168.203.129@tcp15 [19876.247086] LNet: Removed LNI 192.168.203.129@tcp16 [19876.249362] LNet: Removed LNI 192.168.203.129@tcp17 [19876.255178] LNet: Removed LNI 192.168.203.129@tcp18 [19876.261527] LNet: Removed LNI 192.168.203.129@tcp19 [19876.267091] LNet: Removed LNI 192.168.203.129@tcp20 [19876.276474] LNet: Removed LNI 192.168.203.129@tcp21 [19876.289063] LNet: Removed LNI 192.168.203.129@tcp22 [19876.292813] LNet: Removed LNI 192.168.203.129@tcp23 [19876.300274] LNet: Removed LNI 192.168.203.129@tcp24 [19876.302015] LNet: Removed LNI 192.168.203.129@tcp25 [19876.311831] LNet: Removed LNI 192.168.203.129@tcp26 [19876.321994] LNet: Removed LNI 192.168.203.129@tcp27 [19876.328879] LNet: Removed LNI 192.168.203.129@tcp28 [19876.333570] LNet: Removed LNI 192.168.203.129@tcp29 [19876.340083] LNet: Removed LNI 192.168.203.129@tcp30 [19876.341955] LNet: Removed LNI 192.168.203.129@tcp31 [19876.349048] LNet: Removed LNI 192.168.203.129@tcp32 [19876.350658] LNet: Removed LNI 192.168.203.129@tcp33 [19876.369023] LNet: Removed LNI 192.168.203.129@tcp34 [19876.373726] LNet: Removed LNI 192.168.203.129@tcp35 [19876.375547] LNet: Removed LNI 192.168.203.129@tcp36 [19876.387478] LNet: Removed LNI 192.168.203.129@tcp37 [19876.390484] LNet: Removed LNI 192.168.203.129@tcp38 [19876.395926] LNet: Removed LNI 192.168.203.129@tcp39 [19876.397648] LNet: Removed LNI 192.168.203.129@tcp40 [19876.406927] LNet: Removed LNI 192.168.203.129@tcp41 [19876.411217] LNet: Removed LNI 192.168.203.129@tcp42 [19876.415831] LNet: Removed LNI 192.168.203.129@tcp43 [19876.433682] LNet: Removed LNI 192.168.203.129@tcp44 [19876.435520] LNet: Removed LNI 192.168.203.129@tcp45 [19876.448116] LNet: Removed LNI 192.168.203.129@tcp46 [19876.449763] LNet: Removed LNI 192.168.203.129@tcp47 [19876.457877] LNet: Removed LNI 192.168.203.129@tcp48 [19876.463085] LNet: Removed LNI 192.168.203.129@tcp49 [19876.480818] LNet: Removed LNI 192.168.203.129@tcp50 [19877.801146] Key type .llcrypt unregistered [19877.804289] Key type ._llcrypt unregistered [19905.737500] Key type ._llcrypt registered [19905.742218] Key type .llcrypt registered [19905.865678] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [19933.127469] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [19934.767910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [19934.822775] alg: No test for adler32 (adler32-zlib) [19936.460831] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [19936.968119] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [19938.864987] Key type lgssc registered [19941.811761] Lustre: Echo OBD driver; http://www.lustre.org/ [20000.252506] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20000.273649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20001.802142] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20001.894587] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20002.101483] Lustre: lustre-MDT0000: new disk, initializing [20002.369404] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20002.458316] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20009.423762] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [20025.714796] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20025.937888] Lustre: 506886:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20026.020386] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20026.024761] Lustre: Skipped 1 previous similar message [20026.147655] Lustre: lustre-MDT0001: new disk, initializing [20026.266477] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20026.323805] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20026.349983] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20033.683416] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [20047.649396] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20054.505380] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20064.411693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20064.720549] Lustre: *** cfs_fail_loc=90f, val=0*** [20064.789522] Lustre: lustre-OST0000: new disk, initializing [20064.796917] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20064.819483] Lustre: 508906:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20064.955285] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20073.536532] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [20073.575462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20073.588748] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20073.721928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20086.984642] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20099.557984] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [20099.566635] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20099.599731] Lustre: Skipped 1 previous similar message [20099.603115] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20104.675083] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20104.694311] Lustre: Skipped 2 previous similar messages [20105.938764] Lustre: server umount lustre-OST0000 complete [20113.377311] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20113.393263] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20113.425270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20114.920266] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20117.755323] Lustre: server umount lustre-MDT0000 complete [20124.712676] LustreError: 507690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785470801 with bad export cookie 10541460597968086001 [20124.752018] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20124.756101] LustreError: 507690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [20126.377732] Lustre: server umount lustre-MDT0001 complete [20149.541584] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [20154.271978] Key type lgssc unregistered [20154.729105] LNet: 510835:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20154.745781] LNetError: 510835:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20154.773954] LNet: Removed LNI 192.168.203.129@tcp [20156.396822] Key type .llcrypt unregistered [20156.402121] Key type ._llcrypt unregistered [20184.540301] Key type ._llcrypt registered [20184.541729] Key type .llcrypt registered [20184.660705] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [20186.019447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20186.057226] alg: No test for adler32 (adler32-zlib) [20187.337536] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [20187.776632] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [20189.594860] Key type lgssc registered [20191.029724] Lustre: Echo OBD driver; http://www.lustre.org/ [20202.607952] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20202.639278] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20204.638298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20213.682818] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [20214.755405] LustreError: 512107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20214.795815] LustreError: 512107:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [20219.874579] LustreError: 512108:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20224.997971] LustreError: 512107:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20229.238055] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20229.699886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20235.996936] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [20247.323429] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20255.083279] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20263.848306] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20264.307428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20273.515566] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [20285.039803] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20298.538467] loop0: detected capacity change from 0 to 204800000 [20298.803785] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20298.817213] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20298.837137] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20306.434260] loop0: detected capacity change from 0 to 204800000 [20306.815561] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20306.838992] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20306.855379] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [20313.323343] loop0: detected capacity change from 0 to 204800000 [20320.811207] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20321.036790] Lustre: fs15246-OST0000: new disk, initializing [20321.047593] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [20321.063540] Lustre: 515703:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20321.155286] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20329.233343] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [20341.206932] loop1: detected capacity change from 0 to 204800000 [20348.536100] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20348.740527] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [20348.803136] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [20348.897641] Lustre: fs15246-MDT0000: new disk, initializing [20349.061115] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20349.082232] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20349.229790] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [20350.242507] Lustre: 516828:0:(client.c:1639:after_reply()) @@@ resending request on EINPROGRESS req@00000000b52d4066 x1872201896746624/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1785471042 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [20354.740318] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [20359.470861] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [20359.485788] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [20359.528237] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [20383.738466] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20383.763929] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [20388.135448] Lustre: server umount fs15246-MDT0000 complete [20388.842752] LustreError: 512103:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20394.018189] LustreError: 512102:0:(ldlm_lib.c:1192:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20403.868972] Lustre: server umount fs15246-OST0000 complete [20419.556278] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20419.577503] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20424.168249] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [20424.178843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20424.207866] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20425.803570] Lustre: server umount lustre-OST0000 complete [20434.409446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [20434.417774] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20434.443275] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20434.445627] Lustre: Skipped 1 previous similar message [20436.670478] Lustre: server umount lustre-MDT0000 complete [20440.033732] LustreError: 514601:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20440.064947] LustreError: 514601:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 2 previous similar messages [20441.948613] LustreError: 512087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785471118 with bad export cookie 3972577657226505972 [20441.952932] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20441.965271] LustreError: 512087:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [20442.629802] Lustre: server umount lustre-MDT0001 complete [20458.361701] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [20462.208891] Key type lgssc unregistered [20462.623195] LNet: 519930:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20462.631067] LNetError: 519930:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20462.671307] LNet: Removed LNI 192.168.203.129@tcp [20464.039088] Key type .llcrypt unregistered [20464.042973] Key type ._llcrypt unregistered [20489.457248] Key type ._llcrypt registered [20489.467026] Key type .llcrypt registered [20489.579708] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [20490.646969] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [20490.689090] alg: No test for adler32 (adler32-zlib) [20491.889926] Lustre: Lustre: Build Version: 2.17.56_1_gdebb130 [20492.190091] LNet: Added LNI 192.168.203.129@tcp [8/256/0/180] [20493.928180] Key type lgssc registered [20495.536382] Lustre: Echo OBD driver; http://www.lustre.org/ [20542.493920] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [20555.027846] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [20613.981601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [20614.037314] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20615.347208] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [20615.389496] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [20615.467905] Lustre: lustre-MDT0000: new disk, initializing [20615.554434] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [20615.567969] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [20621.960743] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [20636.275508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [20636.485733] Lustre: 524406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [20636.602075] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [20636.618112] Lustre: Skipped 1 previous similar message [20636.810467] Lustre: lustre-MDT0001: new disk, initializing [20636.947729] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [20637.008736] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [20637.028267] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [20643.787434] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [20655.882146] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [20663.324432] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [20674.935190] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [20675.312434] Lustre: lustre-OST0000: new disk, initializing [20675.321081] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [20675.340249] Lustre: 526377:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [20675.532300] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [20682.369198] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [20682.392128] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [20682.634540] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [20685.695321] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_default_debug -1 all [20704.562444] Lustre: DEBUG MARKER: oleg329-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [20711.642711] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [20712.040973] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [20718.771851] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [20719.091831] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [20723.695438] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20723.716360] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [20723.745318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [20723.777263] Lustre: Skipped 1 previous similar message [20729.126095] Lustre: server umount lustre-OST0000 complete [20739.043015] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [20739.061286] Lustre: Skipped 1 previous similar message [20739.079133] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20744.167666] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20744.170027] Lustre: Skipped 1 previous similar message [20749.281136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [20749.281549] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 4. Is it stuck? [20749.297718] Lustre: Skipped 1 previous similar message [20750.143761] Lustre: server umount lustre-MDT0000 complete [20754.402698] LustreError: 524417:0:(ldlm_lib.c:1192:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [20754.433161] LustreError: 524417:0:(ldlm_lib.c:1192:target_handle_connect()) Skipped 4 previous similar messages [20755.135801] LustreError: 524398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1785471431 with bad export cookie 15624342742057173126 [20755.144817] LustreError: MGC192.168.203.129@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [20755.144822] LustreError: 524398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [20755.880813] Lustre: server umount lustre-MDT0001 complete [20794.760601] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing set_hostid [20813.406104] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing load_modules_local [20857.005113] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 20466 sec ========= 00:18:51 (1785471531) [20859.601612] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 00:18:53 (1785471533) === [20865.447275] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 00:18:59 (1785471539) === [20919.477913] Lustre: DEBUG MARKER: oleg329-server.virtnet: executing unload_modules_local [20923.588089] Key type lgssc unregistered [20924.113585] LNet: 533053:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [20924.129442] LNetError: 533053:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [20924.147587] LNet: Removed LNI 192.168.203.129@tcp [20926.056770] Key type .llcrypt unregistered [20926.062064] Key type ._llcrypt unregistered