[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 676526325 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002369] x2apic enabled [ 0.003009] Switched APIC routing to physical x2apic. [ 0.004016] kvm-guest: setup PV IPIs [ 0.007316] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008018] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009013] pid_max: default: 32768 minimum: 301 [ 0.010130] LSM: Security Framework initializing [ 0.012051] Yama: becoming mindful. [ 0.013047] SELinux: Initializing. [ 0.014079] *** VALIDATE selinux *** [ 0.021786] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025769] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027132] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028118] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029112] *** VALIDATE tmpfs *** [ 0.031282] *** VALIDATE proc *** [ 0.032267] *** VALIDATE cgroup *** [ 0.034006] *** VALIDATE cgroup2 *** [ 0.035262] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.036164] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.037009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.038031] Spectre V2 : User space: Vulnerable [ 0.039012] Speculative Store Bypass: Vulnerable [ 0.041646] debug: unmapping init [mem 0xffffffff98659000-0xffffffff98660fff] [ 0.043184] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.044720] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.045027] ... version: 2 [ 0.046016] ... bit width: 48 [ 0.047014] ... generic registers: 4 [ 0.048012] ... value mask: 0000ffffffffffff [ 0.049017] ... max period: 00007fffffffffff [ 0.050012] ... fixed-purpose events: 3 [ 0.050976] ... event mask: 000000070000000f [ 0.051280] rcu: Hierarchical SRCU implementation. [ 0.053375] smp: Bringing up secondary CPUs ... [ 0.054617] x86: Booting SMP configuration: [ 0.055024] .... node #0, CPUs: #1 #2 #3 [ 0.059091] smp: Brought up 1 node, 4 CPUs [ 0.061015] smpboot: Max logical packages: 1 [ 0.062017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.137618] node 0 deferred pages initialised in 71ms [ 0.141303] devtmpfs: initialized [ 0.142254] x86/mm: Memory block size: 128MB [ 0.146149] gcov: version magic: 0x41383552 [ 0.150028] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151077] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.152248] pinctrl core: initialized pinctrl subsystem [ 0.153193] [ 0.153815] ************************************************************* [ 0.154041] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.155009] ** ** [ 0.156008] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.157012] ** ** [ 0.158014] ** This means that this kernel is built to expose internal ** [ 0.159010] ** IOMMU data structures, which may compromise security on ** [ 0.160010] ** your system. ** [ 0.161013] ** ** [ 0.162015] ** If you see this message and you are not debugging the ** [ 0.163012] ** kernel, report this immediately to your vendor! ** [ 0.164009] ** ** [ 0.165010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.166015] ************************************************************* [ 0.167608] NET: Registered protocol family 16 [ 0.169452] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.172074] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.175087] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.179050] cpuidle: using governor menu [ 0.180859] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.184541] PCI: Using configuration type 1 for base access [ 0.186118] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.196070] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.197007] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.199120] cryptd: max_cpu_qlen set to 1000 [ 0.203226] ACPI: Added _OSI(Module Device) [ 0.204012] ACPI: Added _OSI(Processor Device) [ 0.205014] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.207014] ACPI: Added _OSI(Processor Aggregator Device) [ 0.211829] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.218293] ACPI: Interpreter enabled [ 0.219056] ACPI: PM: (supports S0 S3 S4 S5) [ 0.220011] ACPI: Using IOAPIC for interrupt routing [ 0.221146] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.224440] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.234000] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.237043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.240029] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.244115] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.250815] acpiphp: Slot [2] registered [ 0.252231] acpiphp: Slot [5] registered [ 0.254172] acpiphp: Slot [6] registered [ 0.256125] acpiphp: Slot [7] registered [ 0.259142] acpiphp: Slot [8] registered [ 0.261129] acpiphp: Slot [9] registered [ 0.263121] acpiphp: Slot [10] registered [ 0.264112] acpiphp: Slot [3] registered [ 0.266128] acpiphp: Slot [4] registered [ 0.267121] acpiphp: Slot [11] registered [ 0.269118] acpiphp: Slot [12] registered [ 0.271112] acpiphp: Slot [13] registered [ 0.272100] acpiphp: Slot [14] registered [ 0.274102] acpiphp: Slot [15] registered [ 0.275129] acpiphp: Slot [16] registered [ 0.277152] acpiphp: Slot [17] registered [ 0.279111] acpiphp: Slot [18] registered [ 0.280161] acpiphp: Slot [19] registered [ 0.282139] acpiphp: Slot [20] registered [ 0.283138] acpiphp: Slot [21] registered [ 0.285165] acpiphp: Slot [22] registered [ 0.287250] acpiphp: Slot [23] registered [ 0.288141] acpiphp: Slot [24] registered [ 0.290142] acpiphp: Slot [25] registered [ 0.291162] acpiphp: Slot [26] registered [ 0.292149] acpiphp: Slot [27] registered [ 0.294134] acpiphp: Slot [28] registered [ 0.295207] acpiphp: Slot [29] registered [ 0.297135] acpiphp: Slot [30] registered [ 0.299131] acpiphp: Slot [31] registered [ 0.300186] PCI host bridge to bus 0000:00 [ 0.302032] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.305048] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.307035] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.310040] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.313038] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.316041] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.318252] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.322114] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.325600] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.336676] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.341570] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.343027] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.346030] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.348032] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.352559] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.356088] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.359042] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.362867] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.368015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.382019] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.388014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.395288] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.407022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.417023] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.437016] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.448727] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.460015] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.466017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.479014] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.488727] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.498014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.510017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.529018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.546881] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.557027] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.565023] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.618021] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.630818] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.644022] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.652015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.673015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.685292] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.695022] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.704026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.730023] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.744079] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.746344] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.749389] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.752406] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.755294] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.759101] iommu: Default domain type: Passthrough [ 0.762427] SCSI subsystem initialized [ 0.764199] ACPI: bus type USB registered [ 0.766105] usbcore: registered new interface driver usbfs [ 0.768092] usbcore: registered new interface driver hub [ 0.770105] usbcore: registered new device driver usb [ 0.772163] pps_core: LinuxPPS API ver. 1 registered [ 0.774017] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.777126] PTP clock support registered [ 0.780052] EDAC MC: Ver: 3.0.0 [ 0.781110] PCI: Using ACPI for IRQ routing [ 0.782844] NetLabel: Initializing [ 0.784011] NetLabel: domain hash size = 128 [ 0.785009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.787079] NetLabel: unlabeled traffic allowed by default [ 0.789168] vgaarb: loaded [ 0.790356] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.791022] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.797000] clocksource: Switched to clocksource kvm-clock [ 0.918310] VFS: Disk quotas dquot_6.6.0 [ 0.920555] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.924343] *** VALIDATE ramfs *** [ 0.926431] *** VALIDATE hugetlbfs *** [ 0.928827] pnp: PnP ACPI init [ 0.931819] pnp: PnP ACPI: found 6 devices [ 0.959127] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.963252] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.965409] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.967460] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.970150] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.972483] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.975334] NET: Registered protocol family 2 [ 0.977926] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.982723] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.986472] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.991519] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.995125] TCP: Hash tables configured (established 65536 bind 65536) [ 0.997951] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 1.001223] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.004418] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 1.007567] NET: Registered protocol family 1 [ 1.011090] RPC: Registered named UNIX socket transport module. [ 1.017224] RPC: Registered udp transport module. [ 1.019347] RPC: Registered tcp transport module. [ 1.021653] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 1.024226] NET: Registered protocol family 44 [ 1.026290] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 1.028904] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 1.031142] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 1.033681] PCI: CLS 0 bytes, default 64 [ 1.038343] Unpacking initramfs... [ 2.477776] debug: unmapping init [mem 0xffff904ffcc54000-0xffff904ffffbffff] [ 2.481386] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.483830] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.487282] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.992910] Initialise system trusted keyrings [ 2.995035] Key type blacklist registered [ 2.997080] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 3.006987] zbud: loaded [ 3.010313] *** VALIDATE nfs *** [ 3.011669] *** VALIDATE nfs4 *** [ 3.013440] pstore: using deflate compression [ 3.016929] Platform Keyring initialized [ 3.121917] NET: Registered protocol family 38 [ 3.123532] Key type asymmetric registered [ 3.124825] Asymmetric key parser 'x509' registered [ 3.126426] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.129316] io scheduler mq-deadline registered [ 3.130769] io scheduler kyber registered [ 3.132511] io scheduler bfq registered [ 3.134398] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.137093] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.139438] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.142074] ACPI: Power Button [PWRF] [ 3.147914] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.154863] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.170781] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.178226] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.193097] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.223305] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.252218] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.257438] Non-volatile memory driver v1.3 [ 3.259097] Linux agpgart interface v0.103 [ 3.292958] virtio_blk virtio1: [vda] 146624 512-byte logical blocks (75.1 MB/71.6 MiB) [ 3.296187] vda: detected capacity change from 0 to 75071488 [ 3.312019] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.314554] vdb: detected capacity change from 0 to 1073741824 [ 3.329189] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.331747] vdc: detected capacity change from 0 to 2621440000 [ 3.348696] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.351181] vdd: detected capacity change from 0 to 2621440000 [ 3.370461] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.373300] vde: detected capacity change from 0 to 4294967296 [ 3.389476] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.392041] vdf: detected capacity change from 0 to 4294967296 [ 3.399320] libphy: Fixed MDIO Bus: probed [ 3.406427] usbcore: registered new interface driver usbserial_generic [ 3.408720] usbserial: USB Serial support registered for generic [ 3.411140] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.415165] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.417056] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.419152] mousedev: PS/2 mouse device common for all mice [ 3.421989] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.423597] rtc_cmos 00:05: RTC can wake from S4 [ 3.428111] rtc_cmos 00:05: registered as rtc0 [ 3.428387] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.429489] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.433920] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.434537] intel_pstate: CPU model not supported [ 3.439502] hid: raw HID events driver (C) Jiri Kosina [ 3.441492] usbcore: registered new interface driver usbhid [ 3.443656] usbhid: USB HID core driver [ 3.444828] drop_monitor: Initializing network drop monitor service [ 3.446503] Initializing XFRM netlink socket [ 3.447929] NET: Registered protocol family 10 [ 3.450462] Segment Routing with IPv6 [ 3.451802] NET: Registered protocol family 17 [ 3.454029] mpls_gso: MPLS GSO support [ 3.460336] RAS: Correctable Errors collector initialized. [ 3.462438] AVX version of gcm_enc/dec engaged. [ 3.464038] AES CTR mode by8 optimization enabled [ 3.542448] sched_clock: Marking stable (3542414463, 0)->(4473305792, -930891329) [ 3.546199] registered taskstats version 1 [ 3.548196] Loading compiled-in X.509 certificates [ 3.550386] zswap: loaded using pool lzo/zbud [ 3.574915] Key type big_key registered [ 3.590604] Key type encrypted registered [ 3.591906] ima: No TPM chip found, activating TPM-bypass! [ 3.593733] ima: Allocated hash algorithm: sha1 [ 3.595427] ima: No architecture policies found [ 3.596526] evm: Initialising EVM extended attributes: [ 3.597793] evm: security.selinux [ 3.598642] evm: security.ima [ 3.599412] evm: security.capability [ 3.600197] evm: HMAC attrs: 0x1 [ 3.603460] rtc_cmos 00:05: setting system clock to 2026-08-27 11:40:09 UTC (1787830809) [ 3.610131] debug: unmapping init [mem 0xffffffff99603000-0xffffffff997fffff] [ 3.613347] debug: unmapping init [mem 0xffffffff98382000-0xffffffff98658fff] [ 3.625133] Write protecting the kernel read-only data: 28672k [ 3.628613] debug: unmapping init [mem 0xffffffff96a03000-0xffffffff96bfffff] [ 3.631290] debug: unmapping init [mem 0xffffffff97314000-0xffffffff973fffff] [ 3.676956] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.687496] systemd[1]: Detected virtualization kvm. [ 3.689708] systemd[1]: Detected architecture x86-64. [ 3.691806] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.717637] systemd[1]: No hostname configured. [ 3.719254] systemd[1]: Set hostname to . [ 3.721366] random: systemd: uninitialized urandom read (16 bytes read) [ 3.723922] systemd[1]: Initializing machine ID from random generator. [ 3.777733] random: ln: uninitialized urandom read (6 bytes read) [ 3.875602] random: systemd: uninitialized urandom read (16 bytes read) [ 3.878791] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ 3.884759] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.889904] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... [ OK ] Reached target Timers. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Swap. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.551915] device-mapper: uevent: version 1.0.3 [ 4.554512] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 5.351861] virtio_net virtio0 ens2: renamed from eth0 [ 5.389379] random: fast init done [ 5.424428] scsi host0: ata_piix [ 5.457280] scsi host1: ata_piix [ 5.459340] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.462120] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.950626] dracut-initqueue[599]: RTNETLINK answers: File exists [ 10.110215] random: crng init done [ 10.111687] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.668830] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.808946] printk: systemd: 26 output lines suppressed due to ratelimiting [ 12.068352] SELinux: Disabled at runtime. [ 12.129583] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 12.139136] systemd[1]: Detected virtualization kvm. [ 12.141071] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.702240] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.705691] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.715396] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.719289] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.722901] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.731949] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.738856] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting Huge Pages File System... [ OK ] Created slice system-getty.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Stopped target Switch Root. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ 12.786419] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on Process Core Dump Socket. [ OK ] Created slice User and Session Slice. Starting Remount Root and Kernel File Systems... [ OK ] Listening on udev Control Socket. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. Starting Apply Kernel Variables... [ OK ] Reached target Slices. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 13.124531] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.455675] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.485978] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.616141] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.638758] EDAC sbridge: Ver: 1.1.2 [ 15.495257] Key type dns_resolver registered [ 15.818311] NFS: Registering the id_resolver key type [ 15.820334] Key type id_resolver registered [ 15.821984] Key type id_legacy registered [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Reached target sshd-keygen.target. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg655-server login: [ 47.389708] libcfs: loading out-of-tree module taints kernel. [ 47.411469] Key type ._llcrypt registered [ 47.412960] Key type .llcrypt registered [ 47.466680] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 65.017576] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 66.645422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 66.658813] alg: No test for adler32 (adler32-zlib) [ 68.052734] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 68.819714] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 70.584636] Key type lgssc registered [ 72.652919] Lustre: Echo OBD driver; http://www.lustre.org/ [ 92.155307] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 132.149983] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 144.492377] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 144.535309] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 145.781686] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 145.824903] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 145.961789] Lustre: lustre-MDT0000: new disk, initializing [ 146.062038] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 146.078623] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 151.016995] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 164.770861] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 164.913156] Lustre: 6518:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 164.959740] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 164.964084] Lustre: Skipped 1 previous similar message [ 165.088824] Lustre: lustre-MDT0001: new disk, initializing [ 165.185384] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 165.231375] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 165.243827] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 169.830354] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 174.860943] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 185.683563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 186.010769] Lustre: lustre-OST0000: new disk, initializing [ 186.015846] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 186.030494] Lustre: 8457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 186.120809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.049779] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 196.120224] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 196.132635] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 196.215723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 209.622814] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 209.852750] Lustre: lustre-OST0001: new disk, initializing [ 209.856082] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 209.862262] Lustre: 9529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 209.947904] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.666323] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 216.690476] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 216.795056] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 218.260569] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 230.100314] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 236.749593] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 239.101139] hrtimer: interrupt took 2598657 ns [ 244.244688] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing check_logdir /tmp/testlogs/ [ 250.055418] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing yml_node [ 255.022754] Lustre: DEBUG MARKER: Client: 2.17.57.60 [ 257.976651] Lustre: DEBUG MARKER: MDS: 2.17.57.60 [ 261.341727] Lustre: DEBUG MARKER: OSS: 2.17.57.60 [ 263.113681] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Aug 27 07:44:27 EDT 2026 [ 280.535663] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 282.131437] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 298.464224] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 298.484757] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 298.507215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.095712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 304.103395] Lustre: Skipped 5 previous similar messages [ 309.216779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 309.233960] Lustre: Skipped 3 previous similar messages [ 312.287141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 312.708271] Lustre: server umount lustre-MDT0000 complete [ 316.733381] LustreError: 6512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831122 with bad export cookie 14912921386976974389 [ 316.752752] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 319.456133] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 319.456870] LustreError: 10014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 319.467311] Lustre: Skipped 3 previous similar messages [ 319.481690] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 319.505128] LustreError: 10014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 319.525471] Lustre: Skipped 1 previous similar message [ 321.506432] LustreError: 10014:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 321.525758] LustreError: 10014:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 326.627073] LustreError: 12270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 326.627610] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 326.659888] LustreError: 12270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 326.678938] Lustre: Skipped 3 previous similar messages [ 331.232085] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 331.523737] Lustre: server umount lustre-MDT0001 complete [ 335.924702] Lustre: server umount lustre-OST0000 complete [ 339.563175] Lustre: server umount lustre-OST0001 complete [ 345.747602] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 354.234785] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 389.602351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 389.938711] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 389.969408] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 390.050988] Lustre: lustre-MDT0000: new disk, initializing [ 390.114317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 390.129457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 394.285647] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 404.189504] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 404.354979] Lustre: 15086:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 404.406802] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 404.415196] Lustre: Skipped 1 previous similar message [ 404.508217] Lustre: lustre-MDT0001: new disk, initializing [ 404.681131] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 404.730366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 404.749714] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 408.946123] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 418.071668] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 423.205948] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 428.847741] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 429.107780] Lustre: lustre-OST0000: new disk, initializing [ 429.112248] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 429.119661] Lustre: 17054:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 429.196247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 430.266959] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 430.279962] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 430.379829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 434.562238] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 443.143878] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 447.648838] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 447.853200] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 453.499923] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 453.864175] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 456.165593] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 456.166122] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.183611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 456.189280] Lustre: Skipped 2 previous similar messages [ 462.108185] Lustre: server umount lustre-OST0000 complete [ 466.400369] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 466.403610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 466.411068] LustreError: Skipped 1 previous similar message [ 466.432156] Lustre: Skipped 1 previous similar message [ 472.338427] Lustre: server umount lustre-MDT0000 complete [ 475.782571] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831281 with bad export cookie 14912921386976975922 [ 475.791936] LustreError: 15076:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 475.799556] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 476.120936] Lustre: server umount lustre-MDT0001 complete [ 479.783214] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 07:48:04 (1787831284) [ 482.255969] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 484.123644] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 07:48:08 (1787831288) [ 490.262760] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 490.858523] LustreError: 19462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 490.975829] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 494.972702] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 496.099348] LustreError: 19463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 501.984427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 506.392346] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 513.640662] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 519.276442] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 526.598522] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 533.643209] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 542.019576] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 553.953157] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 553.961909] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 553.977975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 553.980541] Lustre: Skipped 7 previous similar messages [ 557.730141] Lustre: server umount lustre-OST0000 complete [ 563.681094] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 563.690023] LustreError: Skipped 1 previous similar message [ 563.696453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 563.715199] Lustre: Skipped 1 previous similar message [ 567.337307] Lustre: server umount lustre-MDT0000 complete [ 569.315785] LustreError: 21629:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 569.339767] LustreError: 21629:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 570.734728] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831376 with bad export cookie 14912921386976976797 [ 570.736646] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.747338] LustreError: 19444:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 570.972711] Lustre: server umount lustre-MDT0001 complete [ 581.377511] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 584.133647] Key type lgssc unregistered [ 584.448976] LNet: 22999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 584.456161] LNetError: 22999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 584.476137] LNet: Removed LNI 192.168.206.155@tcp [ 585.554620] Key type .llcrypt unregistered [ 585.559165] Key type ._llcrypt unregistered [ 605.289635] Key type ._llcrypt registered [ 605.291744] Key type .llcrypt registered [ 605.377816] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 606.279822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 606.417361] alg: No test for adler32 (adler32-zlib) [ 607.479859] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 607.767310] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 609.487143] Key type lgssc registered [ 610.698524] Lustre: Echo OBD driver; http://www.lustre.org/ [ 618.635199] loop: module loaded [ 618.722419] loop0: detected capacity change from 0 to 204800000 [ 618.949621] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 618.957740] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 619.348945] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 624.397734] loop0: detected capacity change from 0 to 204800000 [ 624.549575] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 624.560249] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 624.567751] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 629.147539] loop0: detected capacity change from 0 to 204800000 [ 634.204878] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 634.224383] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 635.463469] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 635.524449] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 635.643941] Lustre: lustre-MDT0000: new disk, initializing [ 635.731650] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 635.750432] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 639.492145] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 646.744691] loop1: detected capacity change from 0 to 204800000 [ 651.188786] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 651.381289] Lustre: lustre-OST07c6: new disk, initializing [ 651.384607] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 651.392251] Lustre: Skipped 1 previous similar message [ 651.399918] Lustre: 26166:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 651.456913] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 654.194048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 654.203569] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 654.244166] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 656.593890] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 660.826269] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 660.833376] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 660.837679] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 660.841959] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 660.853777] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 660.863514] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 672.182128] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 672.190254] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 672.193332] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 672.198586] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 672.207043] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 672.211693] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 676.906800] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 676.915942] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 676.920889] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 676.929981] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.940419] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 676.946675] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.958427] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 676.969470] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.984675] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 676.994920] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 676.999687] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 677.004073] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.255966] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 258 < left 278, rollback = 2 [ 680.280532] Lustre: 25151:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 680.294150] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 680.307727] Lustre: 25151:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.325470] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 680.340406] Lustre: 25151:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.358807] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/0 [ 680.368680] Lustre: 25151:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.384806] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 680.392037] Lustre: 25151:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 680.394647] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 680.403832] Lustre: 25151:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.220933] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 685.231584] Lustre: 25153:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 685.237054] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 685.243039] Lustre: 25153:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.247969] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 685.262623] Lustre: 25153:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.269333] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 685.277771] Lustre: 25153:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.285407] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 685.294060] Lustre: 25153:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 685.304231] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 685.312386] Lustre: 25153:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 696.421143] Lustre: 26783:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 696.435207] Lustre: 26783:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 696.442880] Lustre: 26783:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 696.451955] Lustre: 26783:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 696.459383] Lustre: 26783:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 696.467222] Lustre: 26783:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 696.473186] Lustre: 26783:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 696.481910] Lustre: 26783:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 696.488644] Lustre: 26783:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 696.494090] Lustre: 26783:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 696.499554] Lustre: 26783:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 696.512481] Lustre: 26783:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.481438] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 746.484143] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 746.514233] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 751.584125] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 756.705434] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 757.215163] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 757.377279] Lustre: server umount lustre-OST07c6 complete [ 767.302740] Lustre: server umount lustre-MDT0000 complete [ 782.983875] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 785.806931] Key type lgssc unregistered [ 786.075769] LNet: 28728:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 786.084296] LNetError: 28728:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 786.117172] LNet: Removed LNI 192.168.206.155@tcp [ 787.038885] Key type .llcrypt unregistered [ 787.041898] Key type ._llcrypt unregistered [ 805.664412] Key type ._llcrypt registered [ 805.670317] Key type .llcrypt registered [ 805.798247] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 806.948799] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 806.970879] alg: No test for adler32 (adler32-zlib) [ 807.991881] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 808.172580] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 809.791898] Key type lgssc registered [ 810.763893] Lustre: Echo OBD driver; http://www.lustre.org/ [ 818.108707] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 818.132081] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 819.676721] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 824.932463] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 829.924317] LustreError: 30001:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 829.935603] LustreError: 30001:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 833.583762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 834.026068] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 838.583812] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 844.772214] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 849.199895] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 854.131261] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 854.401983] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 859.434329] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 864.632437] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 865.996718] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 884.192383] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 884.200561] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 884.211471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.221271] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 885.221666] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 885.230557] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 886.462926] Lustre: server umount lustre-OST0000 complete [ 890.344791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 890.351189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 890.355571] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 890.367471] Lustre: Skipped 2 previous similar messages [ 895.464823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 895.856724] Lustre: server umount lustre-MDT0000 complete [ 899.014238] LustreError: 29980:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831704 with bad export cookie 8631541070618257680 [ 899.025610] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 899.355318] Lustre: server umount lustre-MDT0001 complete [ 907.802891] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 910.078809] Key type lgssc unregistered [ 910.335818] LNet: 33738:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 910.345334] LNetError: 33738:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 911.399453] LNet: Removed LNI 192.168.206.155@tcp [ 912.058918] Key type .llcrypt unregistered [ 912.062051] Key type ._llcrypt unregistered [ 928.933380] Key type ._llcrypt registered [ 928.935388] Key type .llcrypt registered [ 929.020423] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 929.811468] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 929.838191] alg: No test for adler32 (adler32-zlib) [ 930.849932] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 931.055408] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 932.703098] Key type lgssc registered [ 933.682661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 941.341471] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 941.365610] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 942.824725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 946.806722] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 952.291671] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 952.312326] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 955.018952] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 955.376154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 959.767750] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 966.760259] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 972.159802] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 978.436339] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 978.765530] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 985.109439] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 988.042884] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 992.718443] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1003.492267] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1003.510962] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1003.511580] Lustre: Skipped 1 previous similar message [ 1003.511973] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1003.540384] Lustre: Skipped 1 previous similar message [ 1008.542698] Lustre: server umount lustre-OST0000 complete [ 1013.733112] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1013.746157] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1013.759110] Lustre: Skipped 1 previous similar message [ 1018.650158] Lustre: server umount lustre-MDT0000 complete [ 1018.848858] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1018.862843] LustreError: 35007:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1021.697384] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831827 with bad export cookie 359728943302411435 [ 1021.702151] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1021.710446] LustreError: 34989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1022.075086] Lustre: server umount lustre-MDT0001 complete [ 1032.615735] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1035.645654] Key type lgssc unregistered [ 1035.974512] LNet: 38543:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1035.982824] LNetError: 38543:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1035.997745] LNet: Removed LNI 192.168.206.155@tcp [ 1037.016694] Key type .llcrypt unregistered [ 1037.021162] Key type ._llcrypt unregistered [ 1062.721454] Key type ._llcrypt registered [ 1062.723107] Key type .llcrypt registered [ 1062.797584] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1063.348487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1063.413443] alg: No test for adler32 (adler32-zlib) [ 1064.475347] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 1064.603879] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1066.359162] Key type lgssc registered [ 1067.316891] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1074.920589] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1074.947719] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1076.343317] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1080.741213] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1086.948826] LustreError: 39961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1086.982775] LustreError: 39961:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1088.754542] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1089.066466] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1092.654709] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1099.271613] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1106.031420] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1113.587795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1114.121453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1120.332190] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1121.308940] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1127.824077] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1138.659510] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1138.660244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1138.672646] Lustre: Skipped 2 previous similar messages [ 1138.676726] Lustre: Skipped 1 previous similar message [ 1140.944862] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 1143.777804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1143.781700] Lustre: Skipped 2 previous similar messages [ 1144.071937] Lustre: server umount lustre-MDT0000 complete [ 1146.036117] LustreError: 42455:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1147.692817] LustreError: 40694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787831953 with bad export cookie 13551118759121028275 [ 1147.699159] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1147.702517] LustreError: 40694:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1148.896392] LustreError: 39957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1148.899589] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.919450] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1151.155921] LustreError: 39955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1152.932991] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1152.937123] Lustre: Skipped 1 previous similar message [ 1154.003732] Lustre: server umount lustre-MDT0001 complete [ 1174.632428] Lustre: server umount lustre-OST0000 complete [ 1188.625481] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1191.018476] Key type lgssc unregistered [ 1191.262930] LNet: 43828:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1191.275388] LNetError: 43828:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1191.292271] LNet: Removed LNI 192.168.206.155@tcp [ 1191.981164] Key type .llcrypt unregistered [ 1191.983076] Key type ._llcrypt unregistered [ 1212.572697] Key type ._llcrypt registered [ 1212.580693] Key type .llcrypt registered [ 1212.720576] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1213.545349] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1213.573424] alg: No test for adler32 (adler32-zlib) [ 1214.666576] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 1214.884382] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1216.511274] Key type lgssc registered [ 1217.541982] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1224.876351] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1224.898390] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1226.535799] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.020536] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1236.964764] LustreError: 45225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1236.977315] LustreError: 45225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1238.908732] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1239.182734] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1243.636169] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1250.090590] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1254.906366] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1260.881387] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1261.178420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1266.319855] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1271.411882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1272.646750] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1278.949234] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1278.967845] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1279.002435] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1282.023139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1282.023278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1282.027077] Lustre: Skipped 1 previous similar message [ 1282.056906] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1283.401423] Lustre: server umount lustre-OST0000 complete [ 1302.497188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1302.531487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1302.537585] Lustre: Skipped 2 previous similar messages [ 1307.617047] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1307.623315] Lustre: Skipped 1 previous similar message [ 1308.219677] Lustre: server umount lustre-MDT0000 complete [ 1308.641751] LustreError: 45225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1308.656959] LustreError: 45225:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1311.830201] LustreError: 45205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832117 with bad export cookie 13721045697684147004 [ 1311.851958] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1311.861670] LustreError: 45205:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1318.255152] Lustre: server umount lustre-MDT0001 complete [ 1327.280452] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1329.717096] Key type lgssc unregistered [ 1329.966730] LNet: 48947:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1329.976575] LNetError: 48947:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1329.992411] LNet: Removed LNI 192.168.206.155@tcp [ 1330.923550] Key type .llcrypt unregistered [ 1330.929496] Key type ._llcrypt unregistered [ 1353.054629] Key type ._llcrypt registered [ 1353.057047] Key type .llcrypt registered [ 1353.153736] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1354.130179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1354.281355] alg: No test for adler32 (adler32-zlib) [ 1355.405324] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 1355.635581] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1357.303227] Key type lgssc registered [ 1358.580302] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1366.099575] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1366.114141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1367.555498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1371.977983] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1377.767079] LustreError: 50359:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1377.781953] LustreError: 50359:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1380.394467] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1380.660453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1384.511809] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1391.463559] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1397.122188] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1403.680654] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1403.983706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1409.862791] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1413.235675] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1417.801203] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1423.741836] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1426.649806] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1438.387114] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-27 8h04m02s [ 1439.598440] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1442.272161] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1442.281501] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1442.309546] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1443.375505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1443.386246] Lustre: Skipped 5 previous similar messages [ 1446.100555] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 1447.838429] Lustre: server umount lustre-MDT0000 complete [ 1449.444540] LustreError: 51606:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1449.458922] LustreError: 51606:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1451.039379] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1452.213025] LustreError: 50355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.598860] LustreError: 50355:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1455.625936] LustreError: 50355:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1458.343847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1458.447973] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1458.661863] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.717700] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1459.705167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1463.369157] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1466.970815] Lustre: DEBUG MARKER: Wait for df (36701) ... [ 1468.627072] Lustre: DEBUG MARKER: done [ 1474.017764] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1474.024785] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1474.039758] Lustre: Skipped 2 previous similar messages [ 1474.042920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1474.050063] Lustre: Skipped 1 previous similar message [ 1475.047417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1479.360910] Lustre: server umount lustre-OST0000 complete [ 1485.282066] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1485.286629] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1485.292057] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1485.299752] Lustre: Skipped 1 previous similar message [ 1485.304919] Lustre: Skipped 2 previous similar messages [ 1488.787299] Lustre: server umount lustre-MDT0000 complete [ 1492.389497] LustreError: 52988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832298 with bad export cookie 11525582579614026194 [ 1492.404227] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1492.409678] LustreError: 52988:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1492.817664] Lustre: server umount lustre-MDT0001 complete [ 1504.338957] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1507.688592] Key type lgssc unregistered [ 1508.060858] LNet: 55284:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1508.075774] LNetError: 55284:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1508.098444] LNet: Removed LNI 192.168.206.155@tcp [ 1509.065303] Key type .llcrypt unregistered [ 1509.067524] Key type ._llcrypt unregistered [ 1552.204114] Key type ._llcrypt registered [ 1552.205434] Key type .llcrypt registered [ 1552.324493] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1553.694851] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1553.709580] alg: No test for adler32 (adler32-zlib) [ 1554.773571] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 1555.013126] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1556.655348] Key type lgssc registered [ 1557.712544] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1564.142588] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1564.169819] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1565.646743] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1565.690696] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1565.910460] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1571.079991] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1581.605604] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1581.765385] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1581.802496] Lustre: 57499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1582.341252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1587.213654] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1595.176649] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1601.223594] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1607.256086] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1607.441638] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1607.696723] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1613.575798] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1615.469037] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1621.706893] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1628.855607] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-27 8h07m13s [ 1630.550280] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1638.134572] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1641.085337] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1641.228874] LustreError: 57509:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1661.658778] Lustre: lustre-MDT0000: Client 01ed5e55-25ec-40e9-8e63-a37c7f50b886 (at 192.168.206.55@tcp) reconnecting [ 1681.239161] LustreError: 57509:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1683.172864] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1685.718508] Lustre: DEBUG MARKER: done [ 1692.129121] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1692.148609] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1692.183971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1697.326148] Lustre: server umount lustre-OST0000 complete [ 1702.881140] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1702.883316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1702.888652] Lustre: Skipped 2 previous similar messages [ 1702.891680] Lustre: Skipped 1 previous similar message [ 1703.974292] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1703.983269] Lustre: Skipped 3 previous similar messages [ 1706.978853] Lustre: server umount lustre-MDT0000 complete [ 1710.559271] LustreError: 57492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832516 with bad export cookie 12758392284033691504 [ 1710.565750] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1710.567716] LustreError: 57492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1710.876672] Lustre: server umount lustre-MDT0001 complete [ 1721.253024] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 1724.270786] Key type lgssc unregistered [ 1724.605470] LNet: 61834:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1724.624102] LNetError: 61834:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1724.660489] LNet: Removed LNI 192.168.206.155@tcp [ 1725.569084] Key type .llcrypt unregistered [ 1725.571392] Key type ._llcrypt unregistered [ 1772.847440] Key type ._llcrypt registered [ 1772.851406] Key type .llcrypt registered [ 1772.956195] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 1773.693932] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1773.761121] alg: No test for adler32 (adler32-zlib) [ 1774.919905] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 1775.255194] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 1776.927230] Key type lgssc registered [ 1778.019939] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1781.014619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1781.051845] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1782.355143] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1782.373824] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1782.550762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1783.401868] Lustre: Failing over lustre-MDT0000 [ 1783.673875] Lustre: server umount lustre-MDT0000 complete [ 1789.845926] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 08:09:54 (1787832594) [ 1796.262846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1796.661726] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1801.195299] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1809.551092] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1809.621092] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1809.655958] Lustre: 64632:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1809.843606] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1813.242456] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1819.848702] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1826.979677] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1834.374891] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1834.595760] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1834.900685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1840.603358] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1840.832556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1847.976772] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1854.255840] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1861.090821] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1861.100068] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.111654] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1861.600758] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1861.623324] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1861.638364] Lustre: Skipped 1 previous similar message [ 1866.725263] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1866.739475] Lustre: Skipped 2 previous similar messages [ 1866.902568] Lustre: server umount lustre-MDT0000 complete [ 1870.261059] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832676 with bad export cookie 1994528367547211574 [ 1870.267872] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1870.279485] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1870.831383] Lustre: server umount lustre-MDT0001 complete [ 1874.280292] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1878.645982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1887.200330] Lustre: 63559:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832677/real 1787832677] req@00000000026557b0 x1874678336495360/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787832693 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1887.226435] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1887.236326] Lustre: Skipped 1 previous similar message [ 1887.434978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1891.295139] Lustre: 63558:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832681/real 1787832681] req@0000000075b4b7ca x1874678336495488/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787832697 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1892.384173] Lustre: 63558:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832682/real 1787832682] req@00000000159396a5 x1874678336495616/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787832698 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1895.469377] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1badfcfb9a176819 [ 1895.473750] Lustre: MGC192.168.206.155@tcp: Connection restored to 0@lo (at 0@lo) [ 1895.793441] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.901877] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1896.799703] Lustre: 63560:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832686/real 1787832686] req@000000000b8e437a x1874678336495872/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787832702 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1899.330861] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1902.943857] Lustre: 63559:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832692/real 1787832692] req@00000000ecb8c749 x1874678336496000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787832708 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1903.075326] LustreError: 68407:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1903.095666] LustreError: 68407:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 1908.193574] LustreError: 68406:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1908.203794] LustreError: 68406:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1909.532975] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1909.846153] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1914.864262] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1915.845341] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1916.991820] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1924.272423] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1930.178469] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1937.383071] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1937.388151] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1937.399657] Lustre: Skipped 1 previous similar message [ 1937.418586] Lustre: Skipped 2 previous similar messages [ 1942.498903] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1942.516885] Lustre: Skipped 2 previous similar messages [ 1951.199143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1951.364455] Lustre: server umount lustre-MDT0000 complete [ 1952.743114] LustreError: 66482:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1952.760288] LustreError: 66482:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1955.036699] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832760 with bad export cookie 1994528367547213849 [ 1955.048912] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1955.053718] LustreError: 64625:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1955.429759] Lustre: server umount lustre-MDT0001 complete [ 1961.288747] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1969.187611] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1973.216255] Lustre: 63560:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787832763/real 1787832763] req@000000003dd91cae x1874678336525696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787832779 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1973.241242] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1973.252995] Lustre: Skipped 1 previous similar message [ 1979.429043] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x1badfcfb9a176d7c [ 1979.457536] Lustre: MGC192.168.206.155@tcp: Connection restored to 0@lo (at 0@lo) [ 1980.029593] LustreError: 69142:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1980.166788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1980.266554] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1985.445921] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 1988.581562] LustreError: 71094:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1988.601120] LustreError: 71094:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1994.978577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2000.124562] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2000.882112] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2000.891176] Lustre: Skipped 1 previous similar message [ 2007.661398] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2012.911030] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2020.848383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2020.853292] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2020.867976] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.870372] Lustre: Skipped 3 previous similar messages [ 2027.284454] Lustre: server umount lustre-MDT0000 complete [ 2031.589669] LustreError: 71092:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2031.635025] LustreError: 71092:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2032.187502] LustreError: 66488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787832838 with bad export cookie 1994528367547215228 [ 2032.189280] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2032.197820] LustreError: 66488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2032.881403] Lustre: server umount lustre-MDT0001 complete [ 2039.897627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2043.161943] Lustre: DEBUG MARKER: files compared the same [ 2055.701488] Lustre: server umount lustre-OST0000 complete [ 2071.139089] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2073.482710] Key type lgssc unregistered [ 2073.773095] LNet: 74372:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2073.777529] LNetError: 74372:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2073.791632] LNet: Removed LNI 192.168.206.155@tcp [ 2074.359094] Key type .llcrypt unregistered [ 2074.360968] Key type ._llcrypt unregistered [ 2095.715424] Key type ._llcrypt registered [ 2095.718191] Key type .llcrypt registered [ 2095.816514] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2096.654147] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2096.785349] alg: No test for adler32 (adler32-zlib) [ 2097.796935] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2098.005503] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2099.631117] Key type lgssc registered [ 2100.619769] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2107.785282] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2107.841631] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.366823] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2114.423488] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug +malloc all [ 2119.659806] LustreError: 75792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2119.690450] LustreError: 75792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2122.420939] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2122.765627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2126.781303] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug +malloc all [ 2134.264919] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2139.302062] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2145.827277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2146.040906] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2151.931878] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug +malloc all [ 2154.232881] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2159.056814] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2168.801023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2168.808359] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2168.827272] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2169.824119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2169.825767] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2169.830164] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2169.831666] Lustre: Skipped 1 previous similar message [ 2173.337139] Lustre: server umount lustre-OST0000 complete [ 2179.041080] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2179.051199] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2179.074691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2185.191703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2185.203979] Lustre: Skipped 3 previous similar messages [ 2190.308955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2190.315012] Lustre: Skipped 1 previous similar message [ 2191.327429] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2191.580041] Lustre: server umount lustre-MDT0000 complete [ 2194.404467] LustreError: 75792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2194.442133] LustreError: 75792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2195.614581] LustreError: 75773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833001 with bad export cookie 8495649873284573346 [ 2195.626062] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2195.628483] LustreError: 75773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2195.903137] Lustre: server umount lustre-MDT0001 complete [ 2207.420944] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2209.916936] Key type lgssc unregistered [ 2210.235546] LNet: 79330:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2210.244420] LNetError: 79330:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2210.254871] LNet: Removed LNI 192.168.206.155@tcp [ 2211.134264] Key type .llcrypt unregistered [ 2211.139514] Key type ._llcrypt unregistered [ 2233.392755] Key type ._llcrypt registered [ 2233.397282] Key type .llcrypt registered [ 2233.503505] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2234.495400] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2234.519573] alg: No test for adler32 (adler32-zlib) [ 2235.588553] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2235.997883] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2237.703156] Key type lgssc registered [ 2239.013195] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2247.010196] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2247.023122] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2262.561387] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2267.679346] LustreError: 80720:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [ 2294.367748] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2302.101398] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2311.120813] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2318.671126] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2318.767337] Lustre: *** cfs_fail_loc=706, val=0*** [ 2319.107086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.167643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2323.112244] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2324.461674] LustreError: 80747:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2324.489274] LustreError: 80747:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2328.549599] LustreError: 81808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2328.568265] LustreError: 81808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2331.046139] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2331.450392] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2336.143892] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2343.853228] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2349.507338] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2352.099946] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2352.101328] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2352.101424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2352.106221] Lustre: Skipped 1 previous similar message [ 2352.118139] Lustre: Skipped 1 previous similar message [ 2357.226669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2357.231833] Lustre: Skipped 1 previous similar message [ 2357.900176] Lustre: server umount lustre-OST0000 complete [ 2362.336441] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2362.346436] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2362.352773] Lustre: Skipped 1 previous similar message [ 2362.369091] Lustre: Skipped 1 previous similar message [ 2367.455996] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2367.469078] Lustre: Skipped 1 previous similar message [ 2369.736049] Lustre: server umount lustre-MDT0000 complete [ 2372.590237] LustreError: 81812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2372.603794] LustreError: 81812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2373.170983] LustreError: 83380:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833179 with bad export cookie 4965334212157036439 [ 2373.183224] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2373.471566] Lustre: server umount lustre-MDT0001 complete [ 2384.350131] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2386.761082] Key type lgssc unregistered [ 2387.001777] LNet: 84339:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2387.009656] LNetError: 84339:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2387.028891] LNet: Removed LNI 192.168.206.155@tcp [ 2387.758144] Key type .llcrypt unregistered [ 2387.760296] Key type ._llcrypt unregistered [ 2411.766798] Key type ._llcrypt registered [ 2411.769540] Key type .llcrypt registered [ 2411.936946] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2412.910255] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2412.940422] alg: No test for adler32 (adler32-zlib) [ 2414.105656] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2414.296774] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2415.983955] Key type lgssc registered [ 2417.064791] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2424.084419] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2424.123434] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2429.463511] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2436.560217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2437.041741] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2441.079612] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2442.209920] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2442.233862] LustreError: 86447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2446.306178] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2446.344628] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2448.926523] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2449.227338] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2449.244342] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2449.266770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2454.497732] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2454.512759] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2454.530259] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2459.616598] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2459.640388] LustreError: 86451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2462.496424] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2462.505931] Lustre: MGS: Client 11fe72dd-7683-462a-bda9-da2a7f061bac (at 0@lo) reconnecting [ 2462.521929] LustreError: 86461:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2462.784191] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2462.856133] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2466.522684] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2483.173518] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2483.188079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2483.198016] LustreError: Skipped 1 previous similar message [ 2483.201536] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2483.248907] Lustre: Skipped 1 previous similar message [ 2485.172796] Lustre: server umount lustre-OST0000 complete [ 2493.417557] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2493.421270] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2493.442527] Lustre: Skipped 1 previous similar message [ 2493.445217] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2493.445225] Lustre: Skipped 1 previous similar message [ 2495.156772] Lustre: server umount lustre-MDT0000 complete [ 2498.545049] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.557814] LustreError: 86450:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2499.290542] Lustre: server umount lustre-MDT0001 complete [ 2502.574981] Lustre: server umount MGS complete [ 2513.716352] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 08:21:57 (1787833317) [ 2542.488800] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 2550.983852] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2583.074627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2587.652414] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2596.945788] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2597.211696] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2597.274202] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2597.337855] Lustre: lustre-MDT0001: new disk, initializing [ 2597.400913] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2597.420588] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2597.434836] Lustre: 93611:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2597.444121] LustreError: 93611:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2601.154636] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2610.729662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2611.016939] Lustre: lustre-OST0000: new disk, initializing [ 2611.023913] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2611.030624] Lustre: 94377:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2611.111747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.929306] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2617.889797] Lustre: 94721:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000fe45c5c2 x1874679006534528/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787833439 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2626.016253] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2632.206895] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2632.234251] Lustre: MGS: Client 33d068fd-6d5e-48f1-a894-c252c6280a6e (at 0@lo) reconnecting [ 2632.285722] LustreError: 93597:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2632.325408] Lustre: 92881:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2632.366906] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2632.436349] Lustre: lustre-MDT0000: new disk, initializing [ 2632.547539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2632.572255] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2632.586062] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2633.634118] Lustre: 95445:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000008c36b927 x1874679006541184/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787833455 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2636.809567] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2638.707865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2638.718728] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2638.798254] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2664.418560] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2664.420021] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2664.431147] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2664.440631] Lustre: Skipped 1 previous similar message [ 2664.449180] Lustre: Skipped 2 previous similar messages [ 2667.374969] Lustre: server umount lustre-OST0000 complete [ 2674.658625] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2674.662971] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2674.680113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2674.694448] Lustre: Skipped 1 previous similar message [ 2677.375092] Lustre: server umount lustre-MDT0000 complete [ 2679.777845] LustreError: 95774:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2679.797984] LustreError: 95774:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2687.289683] Lustre: server umount lustre-MDT0001 complete [ 2690.353433] Lustre: server umount MGS complete [ 2699.880952] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 08:25:04 (1787833504) [ 2716.125813] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2718.367226] Key type lgssc unregistered [ 2718.687895] LNet: 97944:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2718.697702] LNetError: 97944:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2718.715530] LNet: Removed LNI 192.168.206.155@tcp [ 2719.528288] Key type .llcrypt unregistered [ 2719.530393] Key type ._llcrypt unregistered [ 2734.892443] Key type ._llcrypt registered [ 2734.896284] Key type .llcrypt registered [ 2734.986205] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2736.163903] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2736.211037] alg: No test for adler32 (adler32-zlib) [ 2737.201086] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2737.404209] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2739.095631] Key type lgssc registered [ 2739.978380] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2747.532192] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2747.562639] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2749.065464] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2753.244646] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2758.634041] LustreError: 99169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2758.652729] LustreError: 99169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2761.168136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2761.737135] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2766.257550] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2773.471952] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2778.381364] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2784.380370] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2784.672632] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2789.236925] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2793.898819] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 2795.697683] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2804.200226] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2804.219343] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2804.232690] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2809.314154] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2809.325150] Lustre: Skipped 1 previous similar message [ 2810.216567] Lustre: server umount lustre-OST0000 complete [ 2814.433975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2814.440474] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2814.444606] Lustre: Skipped 2 previous similar messages [ 2814.452737] Lustre: Skipped 2 previous similar messages [ 2819.555598] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.678696] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2824.687201] Lustre: Skipped 2 previous similar messages [ 2828.255138] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2828.396242] Lustre: server umount lustre-MDT0000 complete [ 2829.802280] LustreError: 101130:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.830716] LustreError: 101130:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2832.093587] LustreError: 99149:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833637 with bad export cookie 12853890157736542597 [ 2832.115398] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2832.519268] Lustre: server umount lustre-MDT0001 complete [ 2842.342163] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 2844.843295] Key type lgssc unregistered [ 2845.219865] LNet: 102705:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2845.226938] LNetError: 102705:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2845.262869] LNet: Removed LNI 192.168.206.155@tcp [ 2846.105754] Key type .llcrypt unregistered [ 2846.107431] Key type ._llcrypt unregistered [ 2862.100916] Key type ._llcrypt registered [ 2862.103660] Key type .llcrypt registered [ 2862.215898] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 2863.132166] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2863.174546] alg: No test for adler32 (adler32-zlib) [ 2864.443828] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 2864.716398] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 2866.423162] Key type lgssc registered [ 2867.533580] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2873.909485] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2873.921518] LustreError: 103788:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2873.928539] LustreError: 103791:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2873.941133] LustreError: 103788:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 2873.950454] LustreError: 103788:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2873.961304] LustreError: 103788:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2873.968494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2875.351133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2883.369707] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2883.768637] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2888.622276] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2893.536074] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing lsmod [ 2896.145444] LustreError: 104999:0:(tgt_mount.c:2567:server_fill_super()) cfs_race id 716 sleeping [ 2896.147096] LustreError: 104996:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 waking [ 2896.165695] LustreError: 104999:0:(tgt_mount.c:2567:server_fill_super()) cfs_fail_race id 716 awake: rc=4992 [ 2896.176575] LustreError: 104999:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2896.189758] LustreError: 104999:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2896.202559] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2896.539370] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2899.424245] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2899.437788] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2899.456348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.986522] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2901.988379] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2902.019807] Lustre: Skipped 1 previous similar message [ 2905.774213] Lustre: server umount lustre-MDT0000 complete [ 2907.106098] LustreError: 103828:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2907.126971] LustreError: 103828:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2908.130453] LustreError: 104834:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2908.157648] LustreError: 104834:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2909.812777] LustreError: 103801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833715 with bad export cookie 3151267408008336954 [ 2909.816101] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2909.820575] LustreError: 103801:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2909.858458] Lustre: 103820:0:(mgc_request.c:1900:mgc_process_log()) MGC192.168.206.155@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2910.215189] LustreError: 104260:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000837cd936 x1874679478373632/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2910.421737] Lustre: server umount lustre-MDT0001 complete [ 2924.623762] Lustre: server umount lustre-OST0000 complete [ 2933.493892] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2934.044696] LustreError: 106062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2934.122816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2939.200514] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2939.366597] LustreError: 106063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2943.467857] LustreError: 106062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2948.473594] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2948.888849] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2953.775278] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2960.826184] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2966.527169] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2973.632719] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2973.991180] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2979.144855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2980.247731] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 2987.681696] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2998.761038] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2998.762933] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2998.769524] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2998.769534] Lustre: Skipped 2 previous similar messages [ 2998.805450] Lustre: Skipped 1 previous similar message [ 3003.880708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3003.898791] Lustre: Skipped 1 previous similar message [ 3004.943200] Lustre: server umount lustre-OST0000 complete [ 3008.993538] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3009.006984] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3009.011541] Lustre: Skipped 1 previous similar message [ 3009.025073] Lustre: Skipped 2 previous similar messages [ 3014.937813] Lustre: server umount lustre-MDT0000 complete [ 3018.424197] LustreError: 106042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833824 with bad export cookie 3151267408008337647 [ 3018.426397] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3018.442307] LustreError: 106042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3019.072124] Lustre: server umount lustre-MDT0001 complete [ 3031.231881] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3034.972456] Key type lgssc unregistered [ 3035.219572] LNet: 109597:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3035.230727] LNetError: 109597:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3036.268746] LNet: Removed LNI 192.168.206.155@tcp [ 3037.529923] Key type .llcrypt unregistered [ 3037.534071] Key type ._llcrypt unregistered [ 3064.072941] Key type ._llcrypt registered [ 3064.079069] Key type .llcrypt registered [ 3064.185510] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3065.064202] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3065.078451] alg: No test for adler32 (adler32-zlib) [ 3066.170687] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 3066.464316] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3068.175175] Key type lgssc registered [ 3069.674062] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3077.316788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3077.348455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3078.794336] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3082.905482] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3088.365489] LustreError: 111018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3088.393114] LustreError: 111018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3092.416832] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3092.860644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3097.438557] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3104.567716] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3110.052211] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3116.607990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3117.085251] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3123.250587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3123.549184] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3130.917921] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3136.890374] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3140.194165] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3144.360809] Lustre: server umount lustre-OST0000 complete [ 3148.771559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3148.779973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3148.787289] Lustre: Skipped 1 previous similar message [ 3148.801978] Lustre: Skipped 1 previous similar message [ 3153.891611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3153.906069] Lustre: Skipped 1 previous similar message [ 3159.008365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3159.096576] Lustre: server umount lustre-MDT0000 complete [ 3163.632541] LustreError: 111000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787833969 with bad export cookie 3116230738823616638 [ 3163.644663] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3163.648902] LustreError: 111000:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3174.150590] Lustre: server umount lustre-MDT0001 complete [ 3184.914355] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3187.569522] Key type lgssc unregistered [ 3187.913601] LNet: 114663:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3187.922969] LNetError: 114663:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3187.955366] LNet: Removed LNI 192.168.206.155@tcp [ 3188.859701] Key type .llcrypt unregistered [ 3188.861277] Key type ._llcrypt unregistered [ 3205.443680] Key type ._llcrypt registered [ 3205.446173] Key type .llcrypt registered [ 3205.556556] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3206.510965] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3206.542194] alg: No test for adler32 (adler32-zlib) [ 3207.748818] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 3207.961619] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3209.640953] Key type lgssc registered [ 3210.692860] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3218.075277] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3218.110798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3219.708067] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3224.632296] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3229.156869] LustreError: 115885:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3229.169288] LustreError: 115885:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3233.519357] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3233.871672] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3239.151344] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3247.689350] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3253.922187] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3260.926699] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3261.305763] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3264.378099] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3268.517657] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3276.704795] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3289.057963] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3289.066107] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3289.073849] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3289.095716] Lustre: Skipped 1 previous similar message [ 3294.177720] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3294.195199] Lustre: Skipped 2 previous similar messages [ 3294.534633] Lustre: server umount lustre-OST0000 complete [ 3299.298949] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3299.303330] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3299.319768] Lustre: Skipped 1 previous similar message [ 3304.420187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3304.434932] Lustre: Skipped 2 previous similar messages [ 3305.055821] Lustre: server umount lustre-MDT0000 complete [ 3308.875393] LustreError: 115867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834114 with bad export cookie 12607324695350926435 [ 3308.887557] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3309.318073] Lustre: server umount lustre-MDT0001 complete [ 3320.560528] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3323.306637] Key type lgssc unregistered [ 3323.614662] LNet: 119431:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3323.629339] LNetError: 119431:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3324.646869] LNet: Removed LNI 192.168.206.155@tcp [ 3325.557574] Key type .llcrypt unregistered [ 3325.563082] Key type ._llcrypt unregistered [ 3347.116333] Key type ._llcrypt registered [ 3347.121277] Key type .llcrypt registered [ 3347.234720] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3348.432808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3348.487074] alg: No test for adler32 (adler32-zlib) [ 3349.609421] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 3349.928291] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3351.615184] Key type lgssc registered [ 3352.535553] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3359.945901] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3359.963082] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.428389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3365.202542] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3372.007307] LustreError: 120841:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3372.036211] LustreError: 120841:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3373.846283] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3374.188737] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3378.065576] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3385.990246] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3391.677933] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3398.967565] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3399.287892] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3404.894706] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3406.471320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3413.192947] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3422.751409] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3426.461436] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3426.473413] Lustre: Skipped 2 previous similar messages [ 3430.656630] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3430.670718] Lustre: Skipped 2 previous similar messages [ 3432.030336] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3432.071803] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3442.927060] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.55@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3442.941470] Lustre: Skipped 2 previous similar messages [ 3449.954872] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.55@tcp 192.168.0.[2,10]@tcp [ 3454.007260] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.55/24@tcp in log lustre-MDT0000 [ 3454.011652] Lustre: Skipped 2 previous similar messages [ 3456.383683] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.55/24@tcp [ 3456.390372] Lustre: Skipped 1 previous similar message [ 3463.143725] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3463.150845] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3463.173334] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3465.185034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3465.185376] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3465.186162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3465.194900] Lustre: Skipped 1 previous similar message [ 3468.467206] Lustre: server umount lustre-OST0000 complete [ 3475.427351] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3475.428942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3475.448871] Lustre: Skipped 1 previous similar message [ 3475.464413] Lustre: Skipped 1 previous similar message [ 3478.495806] Lustre: server umount lustre-MDT0000 complete [ 3480.545922] LustreError: 120840:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3480.570163] LustreError: 120840:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3482.532737] LustreError: 120822:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834288 with bad export cookie 15812822416579444778 [ 3482.548491] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3482.904426] Lustre: server umount lustre-MDT0001 complete [ 3493.413414] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3496.117733] Key type lgssc unregistered [ 3496.451584] LNet: 125683:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3496.464814] LNetError: 125683:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3496.476903] LNet: Removed LNI 192.168.206.155@tcp [ 3497.335285] Key type .llcrypt unregistered [ 3497.337600] Key type ._llcrypt unregistered [ 3528.213793] Key type ._llcrypt registered [ 3528.217247] Key type .llcrypt registered [ 3528.416024] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3529.188862] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3529.342227] alg: No test for adler32 (adler32-zlib) [ 3530.412377] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 3530.731680] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3532.439171] Key type lgssc registered [ 3533.322994] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3540.169487] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3540.211487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3541.628833] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3541.640803] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3541.664078] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.55/24@tcp [ 3545.951771] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3551.203924] LustreError: 127245:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3551.226662] LustreError: 127245:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3554.153200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3554.461327] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3554.490627] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.55/24@tcp [ 3554.532641] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3558.506033] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3564.939462] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3570.493205] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3575.698804] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3575.966676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3581.264353] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3586.251807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3588.570853] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3597.797857] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 3597.802260] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3597.814679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3600.872526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3600.888615] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3600.906444] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3603.758677] Lustre: server umount lustre-OST0000 complete [ 3611.104383] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3611.115922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3611.126804] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3611.135496] Lustre: Skipped 2 previous similar messages [ 3613.751919] Lustre: server umount lustre-MDT0000 complete [ 3617.034484] LustreError: 127227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834422 with bad export cookie 13437713998708626943 [ 3617.038144] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3617.058663] LustreError: 127227:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3617.381728] Lustre: server umount lustre-MDT0001 complete [ 3628.864720] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3632.288579] Key type lgssc unregistered [ 3632.537607] LNet: 130833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3632.543963] LNetError: 130833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3632.561766] LNet: Removed LNI 192.168.206.155@tcp [ 3633.247331] Key type .llcrypt unregistered [ 3633.249337] Key type ._llcrypt unregistered [ 3659.162927] Key type ._llcrypt registered [ 3659.170272] Key type .llcrypt registered [ 3659.286747] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 3676.378953] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 3677.276753] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3677.389863] alg: No test for adler32 (adler32-zlib) [ 3678.487910] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 3678.733374] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 3680.416842] Key type lgssc registered [ 3681.735703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3715.563969] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3715.618895] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3716.813276] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3716.875769] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3717.002640] Lustre: lustre-MDT0000: new disk, initializing [ 3717.073752] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3717.098592] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3721.511786] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3731.784171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3731.892903] Lustre: 134239:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3731.937218] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3731.941818] Lustre: Skipped 1 previous similar message [ 3732.076485] Lustre: lustre-MDT0001: new disk, initializing [ 3732.191577] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3732.222511] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3732.233937] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3735.809503] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3744.255554] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3749.328191] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3755.624158] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3755.874650] Lustre: lustre-OST0000: new disk, initializing [ 3755.880378] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3755.889163] Lustre: 136205:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3755.999636] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3761.597966] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3762.751406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3762.761343] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3762.869756] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3771.162983] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3776.289957] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3776.542803] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3781.441575] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3781.616123] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3783.668916] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3783.672798] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3783.681881] LustreError: Skipped 1 previous similar message [ 3783.685856] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3783.729622] Lustre: Skipped 1 previous similar message [ 3788.779839] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3788.791167] Lustre: Skipped 2 previous similar messages [ 3789.804221] Lustre: server umount lustre-OST0000 complete [ 3793.890117] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3793.893084] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3793.911334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3793.922649] Lustre: Skipped 1 previous similar message [ 3799.010768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3799.022684] Lustre: Skipped 1 previous similar message [ 3799.873252] Lustre: server umount lustre-MDT0000 complete [ 3803.527258] LustreError: 134230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834609 with bad export cookie 8486622823909370526 [ 3803.531039] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3803.540504] LustreError: 134230:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3803.988590] Lustre: server umount lustre-MDT0001 complete [ 3812.488875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3813.032074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3817.897960] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3818.474231] LustreError: 138423:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3818.492466] LustreError: 138423:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3822.564158] LustreError: 138424:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3826.493662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3826.748680] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3831.345339] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3839.022262] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3846.810497] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3854.252892] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3854.781444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3860.849315] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3868.850550] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3873.713459] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3873.886252] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3878.676620] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3878.890619] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3888.183759] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3888.432916] Lustre: lustre-OST0001: new disk, initializing [ 3888.441294] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3888.448310] Lustre: 141796:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3888.541646] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3894.796712] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 3897.434034] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3897.447329] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3897.569221] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3902.086891] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3902.286978] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3906.158766] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3906.344458] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3911.124650] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ecf46080800.ost_server_uuid 50 [ 3912.468385] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ecf46080800.ost_server_uuid in FULL state after 0 sec [ 3923.425815] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3923.430690] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3923.438654] LustreError: Skipped 1 previous similar message [ 3923.446530] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3923.449419] Lustre: Skipped 2 previous similar messages [ 3923.467812] Lustre: Skipped 1 previous similar message [ 3926.892838] Lustre: server umount lustre-OST0001 complete [ 3928.570889] LustreError: 142262:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3928.613794] LustreError: 142262:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3932.641666] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3932.662122] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3932.685827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3933.675791] LustreError: 140243:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3933.689684] LustreError: 140243:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3937.925538] Lustre: server umount lustre-OST0000 complete [ 3943.906036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3943.927525] Lustre: Skipped 2 previous similar messages [ 3948.674117] Lustre: server umount lustre-MDT0000 complete [ 3949.025355] LustreError: 140390:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3949.067696] LustreError: 140390:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3954.123949] LustreError: 138403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787834760 with bad export cookie 8486622823909371408 [ 3954.131078] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3954.139368] LustreError: 138403:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3954.573424] Lustre: server umount lustre-MDT0001 complete [ 3974.778768] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 3977.489525] Key type lgssc unregistered [ 3977.840580] LNet: 144801:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3977.847549] LNetError: 144801:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3977.866536] LNet: Removed LNI 192.168.206.155@tcp [ 3978.967121] Key type .llcrypt unregistered [ 3978.969631] Key type ._llcrypt unregistered [ 4023.698158] Key type ._llcrypt registered [ 4023.701529] Key type .llcrypt registered [ 4023.825295] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4038.215669] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4039.358222] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4039.386195] alg: No test for adler32 (adler32-zlib) [ 4040.541380] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 4040.853557] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 4042.535151] Key type lgssc registered [ 4043.474695] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4074.364422] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4074.398811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4075.688433] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4075.727528] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4075.826401] Lustre: lustre-MDT0000: new disk, initializing [ 4075.963926] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4075.982445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4079.841806] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4089.107981] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4089.212231] Lustre: 148952:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4089.255852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4089.259266] Lustre: Skipped 1 previous similar message [ 4089.336455] Lustre: lustre-MDT0001: new disk, initializing [ 4089.402989] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4089.434373] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4089.440776] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4093.198428] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4101.047977] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4105.283235] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4111.065888] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4111.245828] Lustre: lustre-OST0000: new disk, initializing [ 4111.248865] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4111.256736] Lustre: 150919:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4111.330337] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4116.459978] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4118.045110] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4118.052977] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4118.114425] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4124.446530] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4128.704823] Lustre: Failing over lustre-OST0000 [ 4128.830411] Lustre: server umount lustre-OST0000 complete [ 4145.699107] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4145.924975] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4145.957953] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4145.982227] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4147.435653] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4147.679249] Lustre: 147177:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834937/real 1787834937] req@000000009f6948d0 x1874680711511680/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1787834953 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4149.727407] Lustre: 147177:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834939/real 1787834939] req@00000000bc73a8a0 x1874680711512832/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787834955 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4151.309595] Lustre: lustre-OST0000: Recovery over after 0:04, of 3 clients 3 recovered and 0 were evicted. [ 4151.310309] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4151.345734] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4154.337558] Lustre: Failing over lustre-MDT0000 [ 4154.593994] Lustre: server umount lustre-MDT0000 complete [ 4155.360256] Lustre: 147177:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834944/real 1787834944] req@0000000044ed2143 x1874680711513856/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787834960 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4155.382681] Lustre: 147177:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 3 previous similar messages [ 4155.875404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4155.883177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4155.900390] Lustre: Skipped 1 previous similar message [ 4159.455469] Lustre: 147174:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787834949/real 1787834949] req@000000000f24e3d9 x1874680711515008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787834965 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4161.507902] LustreError: 151270:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4161.522048] LustreError: 151270:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4162.739954] LustreError: 148959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4166.626978] LustreError: 148963:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4166.655765] LustreError: 148963:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4171.749400] LustreError: 148958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4171.767044] LustreError: 148958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4172.183369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4172.438400] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4172.724194] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4172.785657] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4172.994936] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4177.697047] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4177.900530] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4177.908906] Lustre: Skipped 1 previous similar message [ 4177.929175] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 4177.993990] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4187.104656] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4187.112889] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4187.126032] Lustre: Skipped 2 previous similar messages [ 4187.130827] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4188.133347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4188.143669] Lustre: Skipped 1 previous similar message [ 4193.256789] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4193.271683] Lustre: Skipped 1 previous similar message [ 4198.381724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4198.388457] Lustre: Skipped 1 previous similar message [ 4199.393797] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4199.490665] Lustre: server umount lustre-OST0000 complete [ 4208.608933] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4208.612503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4208.619690] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4208.651065] Lustre: Skipped 2 previous similar messages [ 4218.339723] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4218.547843] Lustre: server umount lustre-MDT0000 complete [ 4218.850731] LustreError: 148962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4218.864922] LustreError: 148962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4221.890790] LustreError: 149756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835027 with bad export cookie 15282726321058599800 [ 4221.896684] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4221.899399] LustreError: 149756:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4222.199648] Lustre: server umount lustre-MDT0001 complete [ 4231.829727] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 4234.499750] Key type lgssc unregistered [ 4234.759590] LNet: 154578:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4234.771205] LNetError: 154578:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4234.796973] LNet: Removed LNI 192.168.206.155@tcp [ 4235.632421] Key type .llcrypt unregistered [ 4235.635097] Key type ._llcrypt unregistered [ 4270.235352] Key type ._llcrypt registered [ 4270.238263] Key type .llcrypt registered [ 4270.370535] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4271.322477] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4271.360484] alg: No test for adler32 (adler32-zlib) [ 4272.475758] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 4272.794057] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 4274.456916] Key type lgssc registered [ 4275.643610] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4282.921549] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4282.944042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4284.330491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4288.233795] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4294.627178] LustreError: 156099:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4294.635377] LustreError: 156099:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4296.086687] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4299.748695] LustreError: 156098:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4299.780222] LustreError: 156127:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4299.792131] LustreError: 156127:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4299.798469] LustreError: 156127:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 4300.050091] Lustre: server umount lustre-MDT0000 complete [ 4317.506857] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4326.275547] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4365.183144] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4365.387528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4365.423875] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4365.550905] Lustre: lustre-MDT0000: new disk, initializing [ 4365.645562] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.658377] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4368.555830] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4377.694664] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4377.802449] Lustre: 159432:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4377.863475] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4377.870558] Lustre: Skipped 1 previous similar message [ 4377.918420] Lustre: lustre-MDT0001: new disk, initializing [ 4378.007686] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4378.049552] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4378.057360] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4382.419743] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4391.773694] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4397.167079] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4402.961915] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4403.293854] Lustre: lustre-OST0000: new disk, initializing [ 4403.298694] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4403.304906] Lustre: 161403:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4403.384755] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4409.607153] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4413.071330] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4413.092245] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4413.195506] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4419.025663] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4423.486724] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4425.110703] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 08:53:49 (1787835229) [ 4679.647719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4679.653479] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4679.656701] Lustre: Skipped 1 previous similar message [ 4679.678808] Lustre: Skipped 2 previous similar messages [ 4683.717931] Lustre: server umount lustre-MDT0000 complete [ 4684.769574] LustreError: 159440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4684.785029] LustreError: 159440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4686.710717] LustreError: 160228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835492 with bad export cookie 12670616414934412877 [ 4686.715783] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4686.720891] LustreError: 160228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4689.898372] LustreError: 159439:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4689.898618] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4689.939761] Lustre: Skipped 1 previous similar message [ 4689.942982] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4691.938165] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4697.056045] LustreError: 160234:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4697.056358] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4697.067981] LustreError: 160234:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4701.152450] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4701.337403] Lustre: server umount lustre-MDT0001 complete [ 4710.608199] Lustre: server umount lustre-OST0000 complete [ 4718.515649] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4727.157648] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4758.110729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4758.389794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4758.414184] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4758.517840] Lustre: lustre-MDT0000: new disk, initializing [ 4758.650891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4758.688660] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4763.082901] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4773.614772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4773.688338] Lustre: 166065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4773.754560] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4773.758192] Lustre: Skipped 1 previous similar message [ 4773.848791] Lustre: lustre-MDT0001: new disk, initializing [ 4773.949251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4774.022339] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4774.036141] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4778.744581] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4788.356199] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4794.235055] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4800.418717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4800.742929] Lustre: lustre-OST0000: new disk, initializing [ 4800.746285] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4800.750451] Lustre: 168036:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4800.819055] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4806.650529] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4807.222529] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4807.247461] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4807.371685] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4815.190841] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4820.097967] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4820.334984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4826.272405] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4826.512316] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4833.261029] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4833.265396] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4833.269919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4833.286165] Lustre: Skipped 1 previous similar message [ 4833.294866] LustreError: Skipped 1 previous similar message [ 4834.638651] Lustre: server umount lustre-OST0000 complete [ 4840.416531] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4840.427845] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4843.490500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4843.493726] Lustre: Skipped 3 previous similar messages [ 4853.215114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4853.383843] Lustre: server umount lustre-MDT0000 complete [ 4853.729249] LustreError: 166949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4853.740156] LustreError: 166949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4857.392051] LustreError: 166860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835663 with bad export cookie 12670616414934571700 [ 4857.394938] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4857.411795] LustreError: 166860:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4857.798912] Lustre: server umount lustre-MDT0001 complete [ 4866.925193] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:01:11 (1787835671) [ 4881.701147] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 4892.581330] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 4935.919563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4936.252787] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4936.288150] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4936.362523] Lustre: lustre-MDT0000: new disk, initializing [ 4936.427154] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4936.458588] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4940.359595] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4949.547851] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4949.699592] Lustre: 172804:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4949.933249] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4953.699453] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4962.610766] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4969.186445] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4975.294530] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4975.634172] Lustre: lustre-OST0000: new disk, initializing [ 4975.641855] Lustre: Skipped 1 previous similar message [ 4975.647158] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4975.653883] Lustre: Skipped 2 previous similar messages [ 4975.659879] Lustre: 174780:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4981.794448] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4981.801635] Lustre: Skipped 1 previous similar message [ 4981.808471] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4981.900590] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4981.988493] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 4993.691938] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5002.723055] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5002.726919] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5002.736776] LustreError: Skipped 1 previous similar message [ 5002.741493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5002.747251] Lustre: Skipped 2 previous similar messages [ 5002.776551] Lustre: Skipped 3 previous similar messages [ 5007.863585] Lustre: server umount lustre-OST0000 complete [ 5011.427640] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5011.434059] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5017.431118] Lustre: server umount lustre-MDT0000 complete [ 5018.081752] LustreError: 172812:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5018.093629] LustreError: 172812:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5020.426449] LustreError: 172795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835826 with bad export cookie 12670616414934572589 [ 5020.434379] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5020.440861] LustreError: 172795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5020.729209] Lustre: server umount lustre-MDT0001 complete [ 5029.046069] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 09:03:52 (1787835832) [ 5049.395052] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 5061.754328] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5095.195814] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5095.478267] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5095.541197] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5095.623798] Lustre: lustre-MDT0000: new disk, initializing [ 5095.683702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5095.688437] Lustre: Skipped 2 previous similar messages [ 5095.705435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5100.054858] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5110.280662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5110.454886] Lustre: 179104:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5110.824918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5115.426917] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5124.569487] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5131.114962] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5137.455249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5137.658883] Lustre: 181081:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5143.346701] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5145.170318] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5145.258318] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5152.902933] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5160.933618] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5160.935284] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5160.941501] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5160.941509] Lustre: Skipped 3 previous similar messages [ 5160.942112] LustreError: Skipped 1 previous similar message [ 5160.951342] Lustre: Skipped 2 previous similar messages [ 5165.519995] Lustre: server umount lustre-OST0000 complete [ 5174.958887] Lustre: server umount lustre-MDT0000 complete [ 5176.290378] LustreError: 180479:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5176.309921] LustreError: 180479:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5178.562422] LustreError: 179904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787835984 with bad export cookie 12670616414934573674 [ 5178.568786] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5178.575051] LustreError: 179904:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5178.975068] Lustre: server umount lustre-MDT0001 complete [ 5192.217639] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5195.729318] Key type lgssc unregistered [ 5196.037694] LNet: 182940:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5196.054870] LNetError: 182940:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5197.093789] LNet: Removed LNI 192.168.206.155@tcp [ 5198.127949] Key type .llcrypt unregistered [ 5198.133723] Key type ._llcrypt unregistered [ 5224.896339] Key type ._llcrypt registered [ 5224.901859] Key type .llcrypt registered [ 5224.981979] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5225.557769] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5225.636939] alg: No test for adler32 (adler32-zlib) [ 5226.634659] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 5226.843835] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 5228.498312] Key type lgssc registered [ 5229.300221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5236.123659] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5236.137584] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5237.427868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5241.476260] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5247.980951] LustreError: 184360:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5248.000552] LustreError: 184360:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5249.052524] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5249.469107] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5253.637580] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5263.041731] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5268.879317] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5276.406637] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5276.814899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5282.256219] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5282.994210] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5288.822500] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5310.949884] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5310.952034] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5310.955176] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5310.975064] Lustre: Skipped 1 previous similar message [ 5310.985699] Lustre: Skipped 1 previous similar message [ 5316.066977] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5316.076337] Lustre: Skipped 1 previous similar message [ 5316.906024] Lustre: server umount lustre-OST0000 complete [ 5321.186130] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5321.226281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5321.243652] Lustre: Skipped 1 previous similar message [ 5326.304473] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5326.318451] Lustre: Skipped 1 previous similar message [ 5327.115138] Lustre: server umount lustre-MDT0000 complete [ 5331.429205] LustreError: 185565:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5331.451040] LustreError: 185565:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5331.467159] LustreError: 184341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836137 with bad export cookie 12645126978439365943 [ 5331.472596] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5331.478849] LustreError: 184341:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5331.946754] Lustre: server umount lustre-MDT0001 complete [ 5345.023298] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 5348.500486] Key type lgssc unregistered [ 5348.771611] LNet: 187918:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5348.781450] LNetError: 187918:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5348.807787] LNet: Removed LNI 192.168.206.155@tcp [ 5349.804294] Key type .llcrypt unregistered [ 5349.807261] Key type ._llcrypt unregistered [ 5376.945344] Key type ._llcrypt registered [ 5376.948585] Key type .llcrypt registered [ 5377.078647] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5378.633896] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5378.683067] alg: No test for adler32 (adler32-zlib) [ 5379.919305] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 5380.303533] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 5382.008166] Key type lgssc registered [ 5383.354814] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5391.920767] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5391.948025] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5393.674548] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5398.907911] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5404.129652] LustreError: 189337:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5404.152561] LustreError: 189337:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5408.603330] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5408.823811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5413.273841] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5421.015280] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5426.511838] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5433.337911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5433.632905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5439.758493] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5439.786648] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5446.746770] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5459.425524] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5459.447919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5459.451545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5459.458651] Lustre: Skipped 1 previous similar message [ 5462.415085] Lustre: server umount lustre-OST0000 complete [ 5468.613942] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9ecf46dac000.ost_server_uuid 50 [ 5469.931230] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 5485.024404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5485.038713] Lustre: Skipped 1 previous similar message [ 5485.042969] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5485.053582] Lustre: Skipped 2 previous similar messages [ 5489.473905] Lustre: server umount lustre-MDT0000 complete [ 5490.144206] LustreError: 190341:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5490.179700] LustreError: 190341:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5492.884869] LustreError: 189318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836298 with bad export cookie 6082552707838972497 [ 5492.891959] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5492.905290] LustreError: 189318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5493.199572] Lustre: server umount lustre-MDT0001 complete [ 5500.574659] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 09:11:45 (1787836305) [ 5506.354907] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5506.741137] LustreError: 193256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5506.805717] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5510.941482] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5512.171914] LustreError: 193257:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5516.256873] LustreError: 193256:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5518.460902] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5518.760868] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5523.079772] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5530.069388] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5536.477309] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5543.555920] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5543.905016] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5549.836479] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5553.131177] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 5557.884715] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5564.977885] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5565.203850] Lustre: lustre-OST0001: new disk, initializing [ 5565.208168] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5565.214269] Lustre: 196096:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5565.292726] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5571.852383] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5574.832117] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5574.853568] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5575.030880] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5581.950802] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5585.392658] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5585.426824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5585.429493] Lustre: Skipped 2 previous similar messages [ 5585.455915] Lustre: Skipped 1 previous similar message [ 5587.704871] Lustre: lustre-OST0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 5592.769373] Lustre: lustre-OST0000: Not available for connect from 192.168.206.55@tcp (stopping) [ 5592.795188] Lustre: Skipped 2 previous similar messages [ 5599.199364] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5599.408923] Lustre: server umount lustre-OST0000 complete [ 5600.753807] LustreError: 196382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5600.789471] LustreError: 196382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5607.247624] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5607.539353] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5612.964978] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5613.137761] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5618.354941] LustreError: 196382:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5618.368654] LustreError: 196382:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 9 previous similar messages [ 5631.472947] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5631.482514] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5631.496237] LustreError: Skipped 1 previous similar message [ 5631.499474] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5631.515992] Lustre: Skipped 1 previous similar message [ 5631.528153] Lustre: Skipped 4 previous similar messages [ 5634.350478] Lustre: server umount lustre-OST0001 complete [ 5641.696342] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5641.704418] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5641.733755] Lustre: Skipped 1 previous similar message [ 5644.987063] Lustre: server umount lustre-MDT0000 complete [ 5649.113278] LustreError: 193238:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836455 with bad export cookie 6082552707838973568 [ 5649.126485] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5649.481415] Lustre: server umount lustre-MDT0001 complete [ 5675.793907] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 09:14:39 (1787836479) [ 5683.374843] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5683.896269] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5683.939171] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5684.076672] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5689.281956] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5698.468294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5698.526721] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5698.556202] Lustre: 199534:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5703.466636] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5711.408944] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5717.221728] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5724.790748] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5724.987771] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5731.269309] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5734.559861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5738.219873] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5744.926982] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5745.035637] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5751.531578] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5758.410122] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5762.031512] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5762.052118] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5762.059093] Lustre: Skipped 3 previous similar messages [ 5767.143887] LustreError: 202422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5767.163180] LustreError: 202422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 5767.294378] Lustre: server umount lustre-OST0000 complete [ 5787.620479] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5787.621858] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5787.650985] Lustre: Skipped 2 previous similar messages [ 5789.340761] Lustre: server umount lustre-OST0001 complete [ 5796.320441] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5796.330944] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5796.344963] Lustre: Skipped 3 previous similar messages [ 5799.334590] Lustre: server umount lustre-MDT0000 complete [ 5803.192140] LustreError: 199525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836609 with bad export cookie 6082552707838974975 [ 5803.199532] LustreError: 199525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5803.208678] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5803.720674] Lustre: server umount lustre-MDT0001 complete [ 5827.382481] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 09:17:11 (1787836631) [ 5843.111911] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 5852.908802] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 5886.847627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5887.152461] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5887.192646] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5887.300486] Lustre: lustre-MDT0000: new disk, initializing [ 5887.427521] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5887.435611] Lustre: Skipped 3 previous similar messages [ 5887.454200] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5891.610801] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5901.797776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5901.945984] Lustre: 207629:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5901.990175] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5901.993536] Lustre: Skipped 1 previous similar message [ 5902.048663] Lustre: lustre-MDT0001: new disk, initializing [ 5902.204652] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5902.222670] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5906.452274] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5915.585312] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5921.111707] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5927.154265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5927.468186] Lustre: lustre-OST0000: new disk, initializing [ 5927.471863] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5927.479041] Lustre: 209597:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5933.454573] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5933.612358] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5933.621159] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5933.704838] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5941.862587] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5946.487414] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5946.702661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5951.247851] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5951.469626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5954.019706] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5954.026582] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5954.027583] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5954.027588] Lustre: Skipped 2 previous similar messages [ 5954.046222] Lustre: Skipped 3 previous similar messages [ 5959.606871] Lustre: server umount lustre-OST0000 complete [ 5970.063893] Lustre: server umount lustre-MDT0000 complete [ 5973.814486] LustreError: 208423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836779 with bad export cookie 6082552707838976536 [ 5973.815471] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5973.820560] LustreError: 208423:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5974.138466] Lustre: server umount lustre-MDT0001 complete [ 5983.262577] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5983.812203] LustreError: 211813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5983.829628] LustreError: 211813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 5988.538173] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 5997.451526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6003.361834] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6011.348039] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6018.618287] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6026.263160] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6026.831072] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6026.835636] Lustre: Skipped 4 previous similar messages [ 6033.517214] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6042.135967] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6047.087535] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6047.322944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6052.163769] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6052.366545] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6054.369585] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6054.387939] LustreError: Skipped 1 previous similar message [ 6054.392733] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6054.420519] Lustre: Skipped 2 previous similar messages [ 6060.472245] Lustre: server umount lustre-OST0000 complete [ 6067.354210] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6073.728695] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6101.812388] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6107.376689] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6107.393889] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6108.473138] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6118.104564] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6122.407566] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6122.727519] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6122.747191] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6128.101054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6128.108392] LustreError: Skipped 1 previous similar message [ 6128.113230] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6128.116585] Lustre: Skipped 11 previous similar messages [ 6132.662696] Lustre: server umount lustre-OST0000 complete [ 6142.474943] Lustre: server umount lustre-MDT0000 complete [ 6143.456521] LustreError: 213760:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6143.493175] LustreError: 213760:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6146.332233] LustreError: 211795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787836952 with bad export cookie 6082552707838977418 [ 6146.347377] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6146.349794] LustreError: 211795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6146.651807] Lustre: server umount lustre-MDT0001 complete [ 6155.043901] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 09:22:39 (1787836959) [ 6161.471549] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6165.655274] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6173.074751] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6178.414635] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6185.605555] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6192.044937] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6197.706882] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6203.233461] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6210.702887] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6216.855790] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6217.164812] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6222.467847] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6222.671427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6229.615961] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6229.767287] Lustre: lustre-OST0001: new disk, initializing [ 6229.770334] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6229.774872] Lustre: 221152:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6235.317201] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6235.326353] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6235.518899] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6238.320167] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6249.432091] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6256.359737] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6256.645162] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6262.460553] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6262.817083] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6266.348917] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6266.358206] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6266.359127] LustreError: Skipped 3 previous similar messages [ 6266.384207] Lustre: Skipped 6 previous similar messages [ 6279.135484] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6279.281648] Lustre: server umount lustre-OST0001 complete [ 6286.703277] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6286.925327] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6292.159751] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6292.333821] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6320.737139] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6320.938565] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6320.949735] Lustre: Skipped 5 previous similar messages [ 6326.048199] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6326.251146] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6326.272188] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6326.275143] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6333.044340] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6334.583061] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6343.654396] Lustre: server umount lustre-OST0001 complete [ 6367.416842] Lustre: server umount lustre-OST0000 complete [ 6377.510393] Lustre: server umount lustre-MDT0000 complete [ 6381.259504] LustreError: 218516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837187 with bad export cookie 6082552707838978384 [ 6381.268912] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6381.278914] LustreError: 218516:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6387.607992] Lustre: server umount lustre-MDT0001 complete [ 6410.330933] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 09:26:54 (1787837214) [ 6417.172975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6417.358368] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6417.380570] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6421.406926] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6428.982212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6429.166483] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6429.221623] Lustre: 226401:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6433.564758] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6442.423879] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6448.391944] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6454.485711] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6454.693519] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6461.333031] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6467.969297] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6476.057981] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6476.113152] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6482.817323] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6489.815695] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6491.164808] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6495.958765] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6496.153171] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6500.416277] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6500.581960] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6506.646334] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9ecf46dad800.ost_server_uuid 50 [ 6508.390582] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9ecf46dad800.ost_server_uuid in IDLE state after 0 sec [ 6509.422712] Lustre: Permanently deactivating lustre-OST0001 [ 6509.434143] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6512.791241] Lustre: Permanently reactivating lustre-OST0001 [ 6521.316381] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6521.329488] LustreError: Skipped 2 previous similar messages [ 6521.337360] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6521.340025] Lustre: Skipped 18 previous similar messages [ 6523.325278] Lustre: server umount lustre-OST0001 complete [ 6526.951190] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6526.959732] LustreError: 228249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6526.967724] Lustre: Skipped 8 previous similar messages [ 6526.990666] LustreError: 228249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 39 previous similar messages [ 6541.283254] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6541.493597] Lustre: server umount lustre-OST0000 complete [ 6551.663668] Lustre: server umount lustre-MDT0000 complete [ 6555.485272] LustreError: 230809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837361 with bad export cookie 6082552707838979672 [ 6555.497023] LustreError: 230809:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6555.499156] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6555.787760] Lustre: server umount lustre-MDT0001 complete [ 6577.363774] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 09:29:41 (1787837381) [ 6584.281650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6584.499894] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6584.530273] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6584.535567] Lustre: Skipped 5 previous similar messages [ 6588.716317] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6596.627352] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6596.706701] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6596.739888] Lustre: 233094:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6601.091251] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6608.340350] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6613.487876] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6620.401133] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6620.640562] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6620.663839] Lustre: Permanently deactivating lustre-OST0000 [ 6620.678121] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6626.194858] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6629.385210] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6633.030322] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6639.406869] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6639.504648] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6645.361927] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6647.472856] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 6652.407328] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6654.294063] Lustre: Permanently reactivating lustre-OST0000 [ 6654.308863] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6654.313688] Lustre: Skipped 2 previous similar messages [ 6657.383522] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 6657.388836] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6657.397446] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6657.402910] Lustre: Skipped 1 previous similar message [ 6657.426204] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6657.439588] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6687.199381] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6687.321329] Lustre: server umount lustre-OST0001 complete [ 6697.234320] Lustre: server umount lustre-OST0000 complete [ 6715.359114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6715.634909] Lustre: server umount lustre-MDT0000 complete [ 6718.921483] LustreError: 233086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837524 with bad export cookie 6082552707838981317 [ 6718.927734] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6718.933769] LustreError: 233086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6719.244982] Lustre: server umount lustre-MDT0001 complete [ 6730.201900] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 6732.520447] Key type lgssc unregistered [ 6732.818552] LNet: 237981:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6732.825541] LNetError: 237981:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6732.845702] LNet: Removed LNI 192.168.206.155@tcp [ 6733.712328] Key type .llcrypt unregistered [ 6733.715577] Key type ._llcrypt unregistered [ 6752.710117] Key type ._llcrypt registered [ 6752.712937] Key type .llcrypt registered [ 6752.804669] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6753.554578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6753.662179] alg: No test for adler32 (adler32-zlib) [ 6754.804665] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 6755.043923] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 6756.703166] Key type lgssc registered [ 6757.877395] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6766.612505] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6766.650811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.967419] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6772.152931] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6777.319747] LustreError: 239428:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6777.338600] LustreError: 239428:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6779.894473] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6779.998881] Lustre: Found index 1 for lustre-MDT0001, updating log [ 6780.022611] Lustre: 239414:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6780.031678] Lustre: Permanently deactivating lustre-MDT0001 [ 6780.044393] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6780.324473] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6783.814526] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6783.820358] LustreError: 239458:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 6783.833914] LustreError: 239458:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 6783.842446] LustreError: 239458:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 16, retries 1, failed: rc = -108 [ 6784.162513] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6790.762216] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6795.399829] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6800.808333] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6801.120568] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6805.257068] LustreError: 241271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6805.269829] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 6807.222339] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6810.600665] LustreError: 241272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.660566] LustreError: 241271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6812.667377] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 6812.689614] LustreError: 241271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6814.169276] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6817.765521] LustreError: 241271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6817.797385] LustreError: 241271:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6820.437500] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6820.739841] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6825.977304] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 6825.990089] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 6827.236925] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6834.239932] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6838.021164] Lustre: Permanently reactivating lustre-MDT0001 [ 6838.041733] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 6838.048278] Lustre: Skipped 2 previous similar messages [ 6843.122960] Lustre: lustre-MDT0001: Client 0b51a13c-b0cf-4663-934a-49fb1be3801c (at 192.168.206.55@tcp) reconnecting [ 6849.472945] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6851.118858] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6856.310820] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6856.524170] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6860.988633] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6861.190851] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6863.661061] Lustre: Permanently deactivating lustre-MDT0001 [ 6863.682697] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6863.689993] Lustre: Skipped 2 previous similar messages [ 6872.684779] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6882.298262] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6882.301604] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6882.319795] Lustre: Skipped 3 previous similar messages [ 6882.331760] Lustre: Skipped 3 previous similar messages [ 6886.495988] Lustre: server umount lustre-MDT0000 complete [ 6887.397084] LustreError: 239422:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6887.420236] LustreError: 239422:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 6890.264368] LustreError: 240169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837696 with bad export cookie 3612170240243924710 [ 6890.265376] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6890.275106] LustreError: 240169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6890.600845] Lustre: server umount lustre-MDT0001 complete [ 6896.683752] Lustre: server umount lustre-OST0000 complete [ 6907.002869] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 09:35:11 (1787837711) [ 6915.352907] Lustre: server umount lustre-OST0001 complete [ 6921.327495] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 6933.743286] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 6969.831499] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6970.054838] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6970.067772] Lustre: Skipped 2 previous similar messages [ 6970.090788] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6970.196180] Lustre: lustre-MDT0000: new disk, initializing [ 6970.333217] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6970.370074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6974.691234] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6984.829502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6984.951690] Lustre: 247731:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6984.998989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6985.003907] Lustre: Skipped 1 previous similar message [ 6985.018452] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 6985.056827] Lustre: lustre-MDT0001: new disk, initializing [ 6985.162753] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6985.202145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6985.215203] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6989.119669] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 6998.378474] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7003.919692] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7011.426095] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7011.909336] Lustre: lustre-OST0000: new disk, initializing [ 7011.913493] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7011.919902] Lustre: 249702:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7012.015540] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7018.621523] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7020.648217] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7020.658558] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7020.739851] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7027.700876] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7029.619246] LustreError: 247737:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7037.844856] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7038.003805] Lustre: lustre-OST0001: new disk, initializing [ 7038.008200] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7038.013597] Lustre: 250818:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7039.688162] LustreError: 247737:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7039.727489] LustreError: 250472:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7044.335887] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7046.722503] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7046.741069] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7046.845618] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7049.783198] LustreError: 250472:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7054.039911] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7055.615864] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7062.500145] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7062.503564] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7062.508154] LustreError: Skipped 1 previous similar message [ 7062.509698] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7062.516881] Lustre: Skipped 1 previous similar message [ 7062.527875] Lustre: Skipped 1 previous similar message [ 7063.609427] Lustre: server umount lustre-OST0001 complete [ 7067.620341] LustreError: 249684:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.639044] LustreError: 249684:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7082.472075] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7082.481547] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7082.497026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7082.508033] Lustre: Skipped 1 previous similar message [ 7088.108773] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7088.121315] Lustre: Skipped 1 previous similar message [ 7094.239333] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7094.417470] Lustre: server umount lustre-OST0000 complete [ 7098.343673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7098.345266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7098.359817] Lustre: Skipped 2 previous similar messages [ 7098.369266] Lustre: Skipped 3 previous similar messages [ 7104.496641] Lustre: server umount lustre-MDT0000 complete [ 7108.203118] LustreError: 247722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787837914 with bad export cookie 3612170240243927524 [ 7108.210057] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7108.222257] LustreError: 247722:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7114.566224] Lustre: server umount lustre-MDT0001 complete [ 7124.274873] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7127.192513] Key type lgssc unregistered [ 7127.498764] LNet: 252853:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7127.512542] LNetError: 252853:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7127.530769] LNet: Removed LNI 192.168.206.155@tcp [ 7128.372555] Key type .llcrypt unregistered [ 7128.380311] Key type ._llcrypt unregistered [ 7167.777874] Key type ._llcrypt registered [ 7167.789335] Key type .llcrypt registered [ 7167.915799] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7169.495910] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7169.577399] alg: No test for adler32 (adler32-zlib) [ 7170.697405] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 7171.008398] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7172.807230] Key type lgssc registered [ 7173.972159] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7181.466250] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7181.477893] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7182.699472] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7182.738594] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7182.902359] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7187.043943] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7196.188319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7196.362185] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7196.421402] Lustre: 255061:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7196.784811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7202.068339] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7210.224929] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7216.216660] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7222.113065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7222.284865] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7222.476086] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7229.344338] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7232.212604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7236.754745] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7288.288650] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7288.300364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7288.335978] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7288.803989] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7288.805169] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7291.951541] Lustre: server umount lustre-OST0000 complete [ 7296.090995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7306.457118] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7306.825676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7308.912653] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7308.930280] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7308.932387] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7308.963850] Lustre: Skipped 1 previous similar message [ 7312.450634] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7320.146777] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7360.481286] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7360.481638] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7360.486818] LustreError: Skipped 1 previous similar message [ 7360.487549] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7360.487554] Lustre: Skipped 1 previous similar message [ 7360.508723] Lustre: Skipped 1 previous similar message [ 7365.533207] Lustre: server umount lustre-OST0000 complete [ 7370.722783] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7370.725457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7370.742468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7370.753885] Lustre: Skipped 1 previous similar message [ 7370.764326] Lustre: Skipped 2 previous similar messages [ 7375.806982] Lustre: server umount lustre-MDT0000 complete [ 7379.900595] LustreError: 255054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838185 with bad export cookie 7096814387800540429 [ 7379.907574] LustreError: 255054:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7379.911733] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7380.264980] Lustre: server umount lustre-MDT0001 complete [ 7394.938505] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7397.779425] Key type lgssc unregistered [ 7398.008052] LNet: 260485:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7398.020093] LNetError: 260485:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7398.040735] LNet: Removed LNI 192.168.206.155@tcp [ 7399.056360] Key type .llcrypt unregistered [ 7399.060429] Key type ._llcrypt unregistered [ 7422.054626] Key type ._llcrypt registered [ 7422.058995] Key type .llcrypt registered [ 7422.246681] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7423.095596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7423.164162] alg: No test for adler32 (adler32-zlib) [ 7424.161815] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 7424.448056] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7426.111326] Key type lgssc registered [ 7427.436035] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7436.214722] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7436.242602] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7437.678667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7443.055912] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7448.035736] LustreError: 261902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7448.053504] LustreError: 261902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7452.191162] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7452.597454] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7457.075442] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7464.139353] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7470.350682] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7477.730605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7478.077443] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7482.167429] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7484.524789] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7491.707524] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7523.305366] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7523.308443] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7523.337396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7523.344919] Lustre: Skipped 1 previous similar message [ 7526.731906] Lustre: server umount lustre-OST0000 complete [ 7533.538197] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7533.554655] Lustre: Skipped 1 previous similar message [ 7533.557934] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7533.566545] Lustre: Skipped 1 previous similar message [ 7537.568058] Lustre: server umount lustre-MDT0000 complete [ 7538.659224] LustreError: 261898:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7538.672399] LustreError: 261898:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7541.144245] LustreError: 261883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838347 with bad export cookie 6264409706127960832 [ 7541.153200] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7541.157294] LustreError: 261883:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7541.493433] Lustre: server umount lustre-MDT0001 complete [ 7550.876439] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7552.986169] Key type lgssc unregistered [ 7553.215515] LNet: 266211:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7553.225180] LNetError: 266211:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7553.235634] LNet: Removed LNI 192.168.206.155@tcp [ 7553.932521] Key type .llcrypt unregistered [ 7553.934577] Key type ._llcrypt unregistered [ 7570.580477] Key type ._llcrypt registered [ 7570.586406] Key type .llcrypt registered [ 7570.743428] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7571.740332] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7571.801189] alg: No test for adler32 (adler32-zlib) [ 7572.838284] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 7573.143333] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7574.929313] Key type lgssc registered [ 7576.128909] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7583.180090] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7583.203784] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7584.555211] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7588.124652] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7594.981989] LustreError: 267430:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7595.003272] LustreError: 267430:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7595.927396] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7596.295325] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7600.524652] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7607.840346] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7613.034164] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7619.456417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7619.828050] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7625.616825] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7630.080289] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7634.677507] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7650.785586] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7650.788441] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7650.802024] Lustre: Skipped 1 previous similar message [ 7654.688850] Lustre: server umount lustre-OST0000 complete [ 7657.954051] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7657.960107] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7657.968273] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7657.971204] Lustre: Skipped 1 previous similar message [ 7661.025550] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7661.028525] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7666.146615] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7666.157181] Lustre: Skipped 2 previous similar messages [ 7670.241336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7670.246180] Lustre: Skipped 1 previous similar message [ 7672.288215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7672.672214] Lustre: server umount lustre-MDT0000 complete [ 7675.376214] LustreError: 267429:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7675.395678] LustreError: 267429:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7676.236434] LustreError: 267411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838482 with bad export cookie 5708829983830937651 [ 7676.238298] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7676.251488] LustreError: 267411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7676.567575] Lustre: server umount lustre-MDT0001 complete [ 7686.106775] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7688.619389] Key type lgssc unregistered [ 7688.845452] LNet: 271116:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7688.856227] LNetError: 271116:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7688.872956] LNet: Removed LNI 192.168.206.155@tcp [ 7689.624825] Key type .llcrypt unregistered [ 7689.635097] Key type ._llcrypt unregistered [ 7712.112234] Key type ._llcrypt registered [ 7712.116840] Key type .llcrypt registered [ 7712.208781] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7713.182777] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7713.197632] alg: No test for adler32 (adler32-zlib) [ 7714.263489] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 7714.573502] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7716.311146] Key type lgssc registered [ 7717.357675] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7725.133968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7725.165830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7726.647848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7731.373949] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7736.806541] LustreError: 272534:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7736.825099] LustreError: 272534:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7740.509287] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7740.826715] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7745.535918] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7753.607273] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7759.742924] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7765.787335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7766.176094] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7771.314698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7771.573968] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7778.172565] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7806.444239] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7806.447806] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7806.454593] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7806.462929] Lustre: Skipped 1 previous similar message [ 7806.473501] Lustre: Skipped 1 previous similar message [ 7811.557480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7811.572214] Lustre: Skipped 1 previous similar message [ 7812.729358] Lustre: server umount lustre-OST0000 complete [ 7816.672612] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7816.675279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7816.687159] Lustre: Skipped 1 previous similar message [ 7816.707312] Lustre: Skipped 1 previous similar message [ 7821.792376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7821.802759] Lustre: Skipped 1 previous similar message [ 7823.087558] Lustre: server umount lustre-MDT0000 complete [ 7826.917688] LustreError: 273261:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7826.938529] LustreError: 273261:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7827.020424] LustreError: 272513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838632 with bad export cookie 15099944566450196217 [ 7827.035521] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7827.038385] LustreError: 272513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7827.440759] Lustre: server umount lustre-MDT0001 complete [ 7837.424344] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7839.955835] Key type lgssc unregistered [ 7840.295879] LNet: 276842:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7840.303896] LNetError: 276842:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7841.329348] LNet: Removed LNI 192.168.206.155@tcp [ 7842.103606] Key type .llcrypt unregistered [ 7842.105841] Key type ._llcrypt unregistered [ 7859.362973] Key type ._llcrypt registered [ 7859.365489] Key type .llcrypt registered [ 7859.523571] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 7860.438345] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7860.499831] alg: No test for adler32 (adler32-zlib) [ 7861.585738] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 7861.879102] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 7863.551262] Key type lgssc registered [ 7864.774989] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7872.681883] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7872.725132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7874.288400] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7878.786083] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7884.771865] LustreError: 278078:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7884.785909] LustreError: 278078:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7886.393515] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7886.754952] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7891.398347] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7899.793473] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7906.726507] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7915.822354] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7916.242090] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7920.344214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7924.327297] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 7933.114162] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7947.232175] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7947.240674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7947.256565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7947.287024] Lustre: Skipped 1 previous similar message [ 7952.355220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7952.374758] Lustre: Skipped 2 previous similar messages [ 7952.560381] Lustre: server umount lustre-OST0000 complete [ 7957.472559] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7957.497146] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7957.510111] Lustre: Skipped 1 previous similar message [ 7962.594021] LustreError: 278080:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7962.612258] LustreError: 278080:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7962.747992] Lustre: server umount lustre-MDT0000 complete [ 7966.709937] LustreError: 278042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838772 with bad export cookie 736301702347931911 [ 7966.714770] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7966.722037] LustreError: 278042:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7967.260984] Lustre: server umount lustre-MDT0001 complete [ 7978.728757] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 7981.185151] Key type lgssc unregistered [ 7981.500491] LNet: 281783:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7981.509435] LNetError: 281783:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7981.525942] LNet: Removed LNI 192.168.206.155@tcp [ 7982.344180] Key type .llcrypt unregistered [ 7982.346855] Key type ._llcrypt unregistered [ 7994.214107] Key type ._llcrypt registered [ 7994.217021] Key type .llcrypt registered [ 7994.329776] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8011.906228] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 8025.924573] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8027.305615] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8027.333549] alg: No test for adler32 (adler32-zlib) [ 8028.472363] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 8028.742917] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8030.463866] Key type lgssc registered [ 8031.539881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8065.674054] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8065.705776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8066.946084] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8067.001754] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8067.126743] Lustre: lustre-MDT0000: new disk, initializing [ 8067.225635] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8067.252429] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8071.489333] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8083.166171] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8083.309633] Lustre: 285333:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8083.388493] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8083.394751] Lustre: Skipped 1 previous similar message [ 8083.498681] Lustre: lustre-MDT0001: new disk, initializing [ 8083.617921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8083.644359] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8083.652861] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8087.757305] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8097.799409] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8103.012588] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8109.430933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8109.785386] Lustre: lustre-OST0000: new disk, initializing [ 8109.789713] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8109.795403] Lustre: 287300:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8109.893250] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8116.181916] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8118.322355] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8118.337636] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8118.424525] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8125.889760] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8130.264905] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8130.479762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8133.947571] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8134.170136] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8139.236326] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8139.238993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8139.239523] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8139.254308] Lustre: Skipped 1 previous similar message [ 8139.279501] Lustre: Skipped 1 previous similar message [ 8142.445026] Lustre: server umount lustre-OST0000 complete [ 8149.474228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8149.476733] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8149.496483] Lustre: Skipped 1 previous similar message [ 8149.514288] Lustre: Skipped 1 previous similar message [ 8152.569844] Lustre: server umount lustre-MDT0000 complete [ 8156.038421] LustreError: 285324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787838961 with bad export cookie 17257841107195699497 [ 8156.052735] LustreError: 285324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8156.054502] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8156.488587] Lustre: server umount lustre-MDT0001 complete [ 8165.345905] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 09:56:09 (1787838969) [ 8172.214739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8172.706358] LustreError: 289711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8172.741595] LustreError: 289711:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8172.866881] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8178.153667] LustreError: 289712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8178.358413] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8182.251548] LustreError: 289711:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8187.377888] LustreError: 289712:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8188.981762] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8189.472692] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8193.947194] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8201.770095] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8208.086799] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8215.568872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8215.948876] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8222.405754] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8230.300083] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8243.691095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8243.705940] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8243.727313] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8244.708825] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8249.657722] Lustre: server umount lustre-OST0000 complete [ 8254.951924] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8254.966968] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8254.967246] Lustre: Skipped 2 previous similar messages [ 8254.986799] Lustre: Skipped 3 previous similar messages [ 8260.073538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8260.082263] Lustre: Skipped 1 previous similar message [ 8267.233431] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8267.431314] Lustre: server umount lustre-MDT0000 complete [ 8269.283631] LustreError: 290440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8269.303127] LustreError: 290440:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8270.687435] LustreError: 289692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839076 with bad export cookie 17257841107195700379 [ 8270.707019] LustreError: 289692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8270.719204] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8270.983922] Lustre: server umount lustre-MDT0001 complete [ 8280.614355] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 8283.234189] Key type lgssc unregistered [ 8283.494920] LNet: 293273:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8283.505125] LNetError: 293273:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8283.526793] LNet: Removed LNI 192.168.206.155@tcp [ 8284.345507] Key type .llcrypt unregistered [ 8284.353892] Key type ._llcrypt unregistered [ 8307.051194] Key type ._llcrypt registered [ 8307.052714] Key type .llcrypt registered [ 8307.142664] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 8322.328692] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8323.113217] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8323.213870] alg: No test for adler32 (adler32-zlib) [ 8324.309336] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 8324.553752] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8326.247795] Key type lgssc registered [ 8327.247980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8366.684032] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8366.705544] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8367.907953] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8367.937767] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8368.018149] Lustre: lustre-MDT0000: new disk, initializing [ 8368.078361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8368.099457] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8371.525468] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8380.506767] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.609628] Lustre: 297095:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8380.661157] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8380.665167] Lustre: Skipped 1 previous similar message [ 8380.753328] Lustre: lustre-MDT0001: new disk, initializing [ 8380.846603] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8380.877366] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8380.883678] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8384.608435] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8392.837112] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8397.227055] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8402.502795] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8402.676284] Lustre: lustre-OST2710: new disk, initializing [ 8402.679403] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8402.684230] Lustre: 299064:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8402.753652] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8407.924796] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8411.709744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8411.722678] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8411.809125] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8416.364610] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8423.223319] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8423.439931] Lustre: lustre-OST03e8: new disk, initializing [ 8423.444665] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8423.450475] Lustre: 300128:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8423.566397] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8429.899209] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8431.202769] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8431.228932] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8431.327391] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8437.752305] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8444.924248] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8445.165220] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8449.509216] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8449.709686] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8453.659870] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8453.853099] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8458.089326] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8458.293029] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8462.309712] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8462.323447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8462.325517] Lustre: Skipped 3 previous similar messages [ 8462.334045] Lustre: Skipped 3 previous similar messages [ 8464.237625] Lustre: server umount lustre-MDT0000 complete [ 8467.275376] LustreError: 297089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839273 with bad export cookie 7112150939416986254 [ 8467.283436] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8467.289974] LustreError: 297089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8467.430246] Lustre: lustre-MDT0001-lwp-OST03e8: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8467.539834] Lustre: server umount lustre-MDT0001 complete [ 8477.002712] Lustre: server umount lustre-OST2710 complete [ 8480.682014] Lustre: server umount lustre-OST03e8 complete [ 8495.136491] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 8503.619810] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8540.467741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8540.597168] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8540.629971] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8540.682728] Lustre: lustre-MDT0000: new disk, initializing [ 8540.728215] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8540.738674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8544.201365] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8553.906546] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8554.062554] Lustre: 305385:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8554.154286] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8554.158917] Lustre: Skipped 1 previous similar message [ 8554.251233] Lustre: lustre-MDT0001: new disk, initializing [ 8554.378322] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8554.411464] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8554.439626] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8558.828476] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8569.122529] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8575.298784] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8582.450394] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8582.785757] Lustre: lustre-OST0000: new disk, initializing [ 8582.796849] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8582.810833] Lustre: 307357:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8582.894453] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8589.073231] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8590.906250] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8590.920680] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8590.982341] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8598.110455] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8602.460435] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8602.653019] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8607.296618] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8607.561840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8611.815935] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8611.816494] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8611.839195] Lustre: Skipped 1 previous similar message [ 8611.859473] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8611.865346] Lustre: Skipped 1 previous similar message [ 8615.839618] Lustre: server umount lustre-OST0000 complete [ 8621.028904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8621.039115] LustreError: Skipped 1 previous similar message [ 8621.045989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8621.062913] Lustre: Skipped 1 previous similar message [ 8621.070205] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8625.536471] Lustre: server umount lustre-MDT0000 complete [ 8627.176596] LustreError: 305394:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8627.193245] LustreError: 305394:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 8628.650142] LustreError: 305379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839434 with bad export cookie 7112150939416987941 [ 8628.655432] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8628.667691] LustreError: 305379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8629.128863] Lustre: server umount lustre-MDT0001 complete [ 8637.440727] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 10:04:01 (1787839441) [ 8638.719834] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8640.526465] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 10:04:04 (1787839444) [ 8643.848913] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8665.250072] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8665.416095] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8665.430195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8665.622259] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8670.647350] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8681.467988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8681.597045] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8681.622465] Lustre: 311230:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8685.985993] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8706.155611] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8706.361930] Lustre: Denying initial registration attempt from nid 192.168.206.155@tcp, specified as failover [ 8706.374816] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8706.383902] LustreError: 312683:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -99 [ 8706.390409] LustreError: 312683:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 8706.395588] LustreError: 312683:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8706.472895] Lustre: server umount lustre-OST0000 complete [ 8706.476527] LustreError: 312683:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8712.673498] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8712.677296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8712.687163] Lustre: Skipped 1 previous similar message [ 8712.690755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8712.695709] Lustre: Skipped 2 previous similar messages [ 8718.818476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8718.822945] Lustre: Skipped 3 previous similar messages [ 8718.896950] Lustre: server umount lustre-MDT0000 complete [ 8721.910342] LustreError: 311222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839527 with bad export cookie 7112150939416988823 [ 8721.919526] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8721.931054] LustreError: 311222:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8722.204631] Lustre: server umount lustre-MDT0001 complete [ 8732.111487] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 8735.116867] Key type lgssc unregistered [ 8735.427855] LNet: 313807:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8735.436954] LNetError: 313807:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8735.463766] LNet: Removed LNI 192.168.206.155@tcp [ 8736.501742] Key type .llcrypt unregistered [ 8736.504754] Key type ._llcrypt unregistered [ 8748.477651] Key type ._llcrypt registered [ 8748.480195] Key type .llcrypt registered [ 8748.574755] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8749.558481] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8749.591838] alg: No test for adler32 (adler32-zlib) [ 8750.570847] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 8750.827989] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8752.487170] Key type lgssc registered [ 8753.684327] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8800.723463] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8808.839979] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8808.856722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8810.115982] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8810.135383] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8810.287800] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8814.661773] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8822.579571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8822.685520] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8822.745225] Lustre: 316570:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8823.100192] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8827.376948] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8834.277367] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8839.523140] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8859.184031] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8859.399718] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8859.562777] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8864.667172] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8872.034087] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8877.040287] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8877.061552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8877.120843] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8877.549477] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8880.860851] Lustre: server umount lustre-OST0000 complete [ 8884.706019] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8884.720533] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8884.736446] Lustre: Skipped 1 previous similar message [ 8884.759721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8884.771168] Lustre: Skipped 1 previous similar message [ 8887.780115] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8887.783268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8887.812712] Lustre: Skipped 1 previous similar message [ 8890.714211] Lustre: server umount lustre-MDT0000 complete [ 8894.566394] LustreError: 316561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839700 with bad export cookie 6067617427026117079 [ 8894.575705] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8894.576055] LustreError: 316561:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8894.840701] Lustre: server umount lustre-MDT0001 complete [ 8904.612971] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 8907.221756] Key type lgssc unregistered [ 8907.579236] LNet: 320148:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8907.601347] LNetError: 320148:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8908.645725] LNet: Removed LNI 192.168.206.155@tcp [ 8909.489478] Key type .llcrypt unregistered [ 8909.491931] Key type ._llcrypt unregistered [ 8929.503962] Key type ._llcrypt registered [ 8929.509331] Key type .llcrypt registered [ 8929.588066] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 8930.319946] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8930.451111] alg: No test for adler32 (adler32-zlib) [ 8931.426941] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 8931.614836] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 8933.303141] Key type lgssc registered [ 8934.258054] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8940.803746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8940.819812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8942.191540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8945.082846] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8951.987145] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8952.355258] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8955.680273] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8961.479441] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8966.470388] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8974.128395] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8974.468260] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8980.074603] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 8987.996156] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9003.488944] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9003.501480] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9003.519088] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9004.525508] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9004.539498] Lustre: Skipped 1 previous similar message [ 9004.539880] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9004.571502] Lustre: Skipped 1 previous similar message [ 9005.943217] Lustre: server umount lustre-MDT0000 complete [ 9008.826418] LustreError: 321540:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9008.856550] LustreError: 321540:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9009.588355] LustreError: 321525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839815 with bad export cookie 8156901042373857138 [ 9009.594167] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9009.604052] LustreError: 321525:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9009.636979] LustreError: 324040:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9009.644178] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9009.665906] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 9009.669900] Lustre: Skipped 1 previous similar message [ 9013.967707] LustreError: 321541:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9014.020620] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.55@tcp (stopping) [ 9016.082087] Lustre: server umount lustre-MDT0001 complete [ 9021.910689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 9031.472929] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9034.575235] LustreError: 323377:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9034.603035] LustreError: 323377:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9034.639260] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9034.702234] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 9038.703502] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9039.778879] LustreError: 325139:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9039.795333] LustreError: 325139:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9044.923718] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9048.012422] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9048.525748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9052.876558] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9053.671537] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9060.597959] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9061.880319] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9066.841478] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9068.686316] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9075.680173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9075.686686] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9075.706306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9075.709456] Lustre: Skipped 1 previous similar message [ 9079.264590] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9080.678238] Lustre: server umount lustre-OST0000 complete [ 9089.504338] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9089.510172] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9089.519807] Lustre: Skipped 2 previous similar messages [ 9089.527213] Lustre: Skipped 2 previous similar messages [ 9091.389428] Lustre: server umount lustre-MDT0000 complete [ 9094.514409] LustreError: 321524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839900 with bad export cookie 8156901042373864712 [ 9094.517971] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9094.529152] LustreError: 321524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9094.902310] Lustre: server umount lustre-MDT0001 complete [ 9106.059928] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 9108.401348] Key type lgssc unregistered [ 9108.720696] LNet: 327815:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9108.725849] LNetError: 327815:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9109.797877] LNet: Removed LNI 192.168.206.155@tcp [ 9110.509222] Key type .llcrypt unregistered [ 9110.513501] Key type ._llcrypt unregistered [ 9129.773108] Key type ._llcrypt registered [ 9129.779225] Key type .llcrypt registered [ 9129.868299] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 9130.894328] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9130.926079] alg: No test for adler32 (adler32-zlib) [ 9131.858126] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 9132.085368] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 9133.722682] Key type lgssc registered [ 9134.748224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9141.999658] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9142.012331] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9143.510945] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9147.916174] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9154.020956] LustreError: 329212:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9154.032498] LustreError: 329212:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9155.953812] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9156.365284] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9160.918638] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9167.891151] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9172.221636] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9177.056706] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9177.064067] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9177.074692] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9179.107170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9179.112394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9180.556836] Lustre: server umount lustre-MDT0000 complete [ 9181.217554] LustreError: 329208:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9181.230772] LustreError: 329208:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9182.783916] LustreError: 329945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787839988 with bad export cookie 17236422489803139599 [ 9182.789617] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9182.793023] LustreError: 329945:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9183.022797] Lustre: server umount lustre-MDT0001 complete [ 9192.007940] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9192.166867] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9192.195446] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9192.348139] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9195.295342] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9201.325575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9201.399063] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9201.421981] Lustre: 331795:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9201.621433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9204.835809] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9210.404720] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9214.707521] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9223.408520] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9223.610599] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 9223.619685] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 9223.628295] LustreError: 333678:0:(tgt_mount.c:2611:server_fill_super()) Unable to start targets: -2 [ 9223.635639] LustreError: 333678:0:(tgt_mount.c:2129:server_put_super()) no obd lustre-OST0000 [ 9223.642794] LustreError: 333678:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9223.706564] Lustre: server umount lustre-OST0000 complete [ 9223.710329] LustreError: 333678:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 9232.748708] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9232.911363] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9233.082905] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9239.007954] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9241.764641] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 9247.375666] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9255.264581] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9255.352466] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 9255.426867] Lustre: lustre-OST0001: new disk, initializing [ 9255.430596] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9255.436643] Lustre: 335220:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9255.536064] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9261.575667] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9261.640900] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9261.652774] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9261.762771] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9267.722876] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9272.297653] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9272.297747] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9272.303586] LustreError: Skipped 1 previous similar message [ 9272.306586] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9272.306594] Lustre: Skipped 1 previous similar message [ 9272.324118] Lustre: Skipped 1 previous similar message [ 9276.720508] Lustre: server umount lustre-OST0001 complete [ 9277.417312] LustreError: 334146:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9277.452383] LustreError: 334146:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9282.531790] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9282.537141] LustreError: 334143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9282.558483] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9282.561012] Lustre: Skipped 1 previous similar message [ 9282.561622] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9282.575583] LustreError: 334143:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9282.590659] Lustre: Skipped 2 previous similar messages [ 9287.655933] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9287.657068] LustreError: 334143:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9287.661679] Lustre: Skipped 1 previous similar message [ 9294.303253] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9294.444600] Lustre: server umount lustre-OST0000 complete [ 9297.890119] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9297.895961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9297.900822] Lustre: Skipped 1 previous similar message [ 9297.903586] Lustre: Skipped 2 previous similar messages [ 9312.223951] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9312.387752] Lustre: server umount lustre-MDT0000 complete [ 9313.252087] LustreError: 331801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9313.270516] LustreError: 331801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9315.636761] LustreError: 331786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840121 with bad export cookie 17236422489803140152 [ 9315.647963] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9315.652373] LustreError: 331786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9315.908893] Lustre: server umount lustre-MDT0001 complete [ 9324.440699] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 9326.530980] Key type lgssc unregistered [ 9326.716707] LNet: 337183:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9326.722707] LNetError: 337183:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9326.735850] LNet: Removed LNI 192.168.206.155@tcp [ 9327.387328] Key type .llcrypt unregistered [ 9327.390109] Key type ._llcrypt unregistered [ 9393.141444] Key type ._llcrypt registered [ 9393.143810] Key type .llcrypt registered [ 9393.338738] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [ 9411.019697] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 9412.187589] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9412.304559] alg: No test for adler32 (adler32-zlib) [ 9413.433812] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 9413.725087] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 9415.407198] Key type lgssc registered [ 9416.729611] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9448.579805] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9448.604598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9449.840821] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9449.884402] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9449.965489] Lustre: lustre-MDT0000: new disk, initializing [ 9450.050896] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9450.094085] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9454.188776] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9463.488740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9463.585386] Lustre: 342118:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9463.638306] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9463.641762] Lustre: Skipped 1 previous similar message [ 9463.696313] Lustre: lustre-MDT0001: new disk, initializing [ 9463.761262] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9463.801476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9463.817287] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9468.723748] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9476.964846] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9482.078205] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9487.913358] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9488.167827] Lustre: lustre-OST0000: new disk, initializing [ 9488.178093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9488.182533] Lustre: 344089:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9488.287291] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9494.117482] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9494.131896] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9494.285564] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9494.293808] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9504.540539] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9510.985348] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9511.294594] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9515.663914] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9515.877758] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9520.105233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9520.112923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9520.125672] Lustre: Skipped 1 previous similar message [ 9520.126502] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9520.140779] Lustre: Skipped 1 previous similar message [ 9525.220186] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9525.230377] Lustre: Skipped 1 previous similar message [ 9530.348471] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9530.361148] Lustre: Skipped 1 previous similar message [ 9532.383115] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9532.555534] Lustre: server umount lustre-OST0000 complete [ 9540.581979] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9540.582696] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9540.596213] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9540.609392] Lustre: Skipped 1 previous similar message [ 9542.567868] Lustre: server umount lustre-MDT0000 complete [ 9546.133978] LustreError: 344088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840352 with bad export cookie 5076351710087792018 [ 9546.138817] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9546.143062] LustreError: 344088:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9546.436848] Lustre: server umount lustre-MDT0001 complete [ 9555.021503] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 10:19:19 (1787840359) [ 9562.271824] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 10:19:26 (1787840366) [ 9571.439067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9571.780367] LustreError: 346846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9571.790722] LustreError: 346846:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9571.842884] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9576.237690] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9576.935523] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9582.054284] LustreError: 346846:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9584.818807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9585.154788] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9588.808847] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9597.588400] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9603.977427] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9610.439651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9610.777479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9616.710454] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9623.272914] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9632.790311] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9634.125542] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9635.274530] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9636.679247] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9638.526659] Lustre: Failing over lustre-MDT0000 [ 9638.815268] Lustre: server umount lustre-MDT0000 complete [ 9639.403030] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9639.405308] LustreError: 346842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9639.419092] Lustre: Skipped 2 previous similar messages [ 9639.441279] LustreError: 346842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9644.513258] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9644.535857] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9654.762453] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9654.774223] LustreError: 346847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 9655.776183] Lustre: 340342:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840445/real 1787840445] req@0000000005462f52 x1874686345549824/t0(0) o400->MGC192.168.206.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787840461 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9655.799449] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9656.969497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9666.022157] LustreError: 340341:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000676b377f x1874686345555584/t0(0) o250->MGC192.168.206.155@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9666.301218] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9666.340662] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9666.739139] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9670.125435] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9671.673943] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9671.698961] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9671.750069] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9677.436458] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9679.070179] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9682.463329] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9684.510420] Lustre: Failing over lustre-MDT0000 [ 9684.910876] Lustre: server umount lustre-MDT0000 complete [ 9687.011763] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9687.031958] Lustre: Skipped 1 previous similar message [ 9687.045416] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9687.047840] LustreError: 346842:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9687.074962] LustreError: 346842:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [ 9702.369955] Lustre: 340344:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787840492/real 1787840492] req@00000000e3729dfb x1874686345569280/t0(0) o400->MGC192.168.206.155@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787840508 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9702.413110] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9704.506865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9712.878258] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9712.938515] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9716.678629] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9717.947147] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9718.259543] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 9718.263882] Lustre: Skipped 2 previous similar messages [ 9718.292216] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 9718.340801] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9725.400785] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9727.114637] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9733.600362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9733.606739] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9733.620420] Lustre: Skipped 4 previous similar messages [ 9733.633712] Lustre: Skipped 2 previous similar messages [ 9738.211890] Lustre: server umount lustre-MDT0000 complete [ 9738.754196] LustreError: 350580:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9738.777542] LustreError: 350580:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 22 previous similar messages [ 9742.657440] LustreError: 346826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840548 with bad export cookie 5076351710087794727 [ 9742.668748] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9743.411341] Lustre: server umount lustre-MDT0001 complete [ 9757.486184] Lustre: server umount lustre-OST0000 complete [ 9772.304839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9776.558981] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9784.507204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9784.779587] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9784.789693] Lustre: Skipped 1 previous similar message [ 9788.962567] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9795.575861] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9800.624338] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9806.433799] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9811.773677] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9818.088590] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9818.991064] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9829.345982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9829.349401] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9829.354768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9829.359666] Lustre: Skipped 1 previous similar message [ 9833.166581] Lustre: server umount lustre-OST0000 complete [ 9842.311207] Lustre: server umount lustre-MDT0000 complete [ 9844.704407] LustreError: 353917:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9844.733802] LustreError: 353917:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9845.391948] LustreError: 353167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840651 with bad export cookie 5076351710087795476 [ 9845.392446] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9845.404167] LustreError: 353167:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [ 9845.683919] Lustre: server umount lustre-MDT0001 complete [ 9854.034743] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [ 9855.529665] Key type lgssc unregistered [ 9855.721926] LNet: 356721:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9855.724309] LNetError: 356721:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9855.733083] LNet: Removed LNI 192.168.206.155@tcp [ 9856.198306] Key type .llcrypt unregistered [ 9856.200889] Key type ._llcrypt unregistered [ 9876.814318] Key type ._llcrypt registered [ 9876.817780] Key type .llcrypt registered [ 9876.894369] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [ 9877.962638] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9877.976947] alg: No test for adler32 (adler32-zlib) [ 9879.039154] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [ 9879.225779] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [ 9880.855156] Key type lgssc registered [ 9881.892736] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9887.379550] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9887.395978] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9888.839261] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9892.382878] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9898.631317] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9898.879287] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9902.685944] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9908.236382] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9912.345381] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9917.871984] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9918.193454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9924.351338] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9934.602600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9935.456829] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9950.176176] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9950.183459] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9950.184115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9950.208464] Lustre: Skipped 2 previous similar messages [ 9955.296890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9955.301317] Lustre: Skipped 4 previous similar messages [ 9955.674402] Lustre: server umount lustre-MDT0000 complete [ 9958.817471] LustreError: 358952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840764 with bad export cookie 10589274152968096034 [ 9958.833755] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9959.099769] Lustre: server umount lustre-MDT0001 complete [ 9973.178593] Lustre: server umount lustre-OST0000 complete [ 9986.177699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9986.523422] LustreError: 361949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9986.539725] LustreError: 361949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9986.590431] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9991.622633] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [ 9991.657398] LustreError: 361950:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9996.776101] LustreError: 361949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9999.661520] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10000.005651] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10003.921981] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10011.963618] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10016.954917] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10022.173593] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10022.420218] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10027.160136] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10031.659397] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [10032.615139] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10045.920919] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10045.933192] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10045.960505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10046.962079] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10051.501238] Lustre: server umount lustre-MDT0000 complete [10052.071651] LustreError: 364253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10052.086644] LustreError: 364253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10055.060026] LustreError: 361930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840860 with bad export cookie 10589274152968097140 [10055.075646] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10055.082530] LustreError: 361930:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [10055.730446] Lustre: server umount lustre-MDT0001 complete [10070.961769] Lustre: server umount lustre-OST0000 complete [10083.254448] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10083.590560] LustreError: 365858:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10083.681574] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10086.996631] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10094.056274] LustreError: 365859:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10094.072702] LustreError: 365859:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10094.204532] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10098.572473] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10105.448567] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10111.129660] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10117.565281] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10117.910765] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10117.919510] Lustre: Skipped 1 previous similar message [10122.641338] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10128.175354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [10129.636932] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10135.526767] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10135.538923] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10135.553027] Lustre: Skipped 1 previous similar message [10135.557986] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10135.562393] Lustre: Skipped 3 previous similar messages [10138.603136] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10140.333169] Lustre: server umount lustre-OST0000 complete [10143.720923] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10143.722203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10143.734504] Lustre: Skipped 2 previous similar messages [10143.755904] Lustre: Skipped 3 previous similar messages [10149.839546] Lustre: server umount lustre-MDT0000 complete [10153.372319] LustreError: 365839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787840959 with bad export cookie 10589274152968098169 [10153.372764] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10153.385208] LustreError: 365839:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10153.766984] Lustre: server umount lustre-MDT0001 complete [10165.212792] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [10167.977516] Key type lgssc unregistered [10168.223619] LNet: 369392:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10168.232864] LNetError: 369392:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10169.260666] LNet: Removed LNI 192.168.206.155@tcp [10170.230394] Key type .llcrypt unregistered [10170.233561] Key type ._llcrypt unregistered [10199.618266] Key type ._llcrypt registered [10199.621108] Key type .llcrypt registered [10199.708621] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10200.517822] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10200.571183] alg: No test for adler32 (adler32-zlib) [10201.565855] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [10201.754540] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10203.407169] Key type lgssc registered [10204.289403] Lustre: Echo OBD driver; http://www.lustre.org/ [10210.522768] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10210.552968] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10210.565365] LustreError: 370833:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [10210.600881] LustreError: 370833:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [10210.606703] LustreError: 370833:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [10210.611580] LustreError: 370833:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [10210.617769] LustreError: 370833:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10217.034168] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [10217.040611] LustreError: 371230:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [10217.053979] LustreError: 371230:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [10217.061657] LustreError: 371230:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [10217.065756] LustreError: 371230:0:(tgt_mount.c:2579:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [10217.071798] LustreError: 371230:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [10233.477944] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [10235.963463] Key type lgssc unregistered [10236.190168] LNet: 372236:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10236.197567] LNetError: 372236:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10236.209875] LNet: Removed LNI 192.168.206.155@tcp [10237.000882] Key type .llcrypt unregistered [10237.002270] Key type ._llcrypt unregistered [10254.586983] Key type ._llcrypt registered [10254.589134] Key type .llcrypt registered [10254.678304] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [10267.836039] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10268.594029] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10268.730498] alg: No test for adler32 (adler32-zlib) [10269.736862] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [10269.936409] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10271.583808] Key type lgssc registered [10272.455186] Lustre: Echo OBD driver; http://www.lustre.org/ [10311.059149] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10311.123155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10312.372531] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10312.414591] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10312.539025] Lustre: lustre-MDT0000: new disk, initializing [10312.685257] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10312.711606] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10316.748635] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10326.390643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10326.472761] Lustre: 375394:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10326.542692] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10326.546799] Lustre: Skipped 1 previous similar message [10326.627261] Lustre: lustre-MDT0001: new disk, initializing [10326.795635] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10326.861500] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10326.877011] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10331.671659] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10341.356204] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10346.941505] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10353.250927] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10353.514972] Lustre: lustre-OST0000: new disk, initializing [10353.518820] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10353.524821] Lustre: 377364:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10353.636329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10360.043814] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10360.396460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10360.408700] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10360.509563] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10368.819151] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10373.575934] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10373.758893] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10377.846434] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10378.005932] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10381.285826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10381.287368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10381.291676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10381.299499] Lustre: Skipped 1 previous similar message [10381.305992] LustreError: Skipped 1 previous similar message [10386.403514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10386.409524] Lustre: Skipped 1 previous similar message [10391.531355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10391.539298] Lustre: Skipped 2 previous similar messages [10394.079105] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10394.371389] Lustre: server umount lustre-OST0000 complete [10398.691413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10398.701509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10398.719876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10401.762333] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10404.234767] Lustre: server umount lustre-MDT0000 complete [10407.836239] LustreError: 375387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841213 with bad export cookie 13306653632527495245 [10407.839453] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10407.843042] LustreError: 375387:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10408.366322] Lustre: server umount lustre-MDT0001 complete [10416.943556] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [10418.535625] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 10:33:42 (1787841222) [10425.333973] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10425.706470] LustreError: 379822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10425.722417] LustreError: 379822:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10425.770777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10430.403065] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10430.946443] LustreError: 379823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10436.070740] LustreError: 379822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10438.741072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10439.033214] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10443.375703] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10452.009919] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10458.110428] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10465.653518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10466.022604] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10472.895380] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10479.099509] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10485.456755] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10485.594302] Lustre: lustre-OST0001: new disk, initializing [10485.599570] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10485.608330] Lustre: 382659:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10485.673065] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10491.516982] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10491.525664] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10491.627246] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10491.961575] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10502.023206] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10507.237141] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10507.252145] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10507.252737] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10507.264934] Lustre: Skipped 3 previous similar messages [10512.360524] LustreError: 382949:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10512.380854] LustreError: 382949:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10512.457929] Lustre: server umount lustre-OST0001 complete [10517.499319] LustreError: 381646:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10517.518491] LustreError: 381646:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10527.724863] LustreError: 381644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10527.750697] LustreError: 381644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10531.815269] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10531.832938] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10531.845839] Lustre: Skipped 1 previous similar message [10531.854547] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10531.861564] Lustre: Skipped 1 previous similar message [10536.838592] Lustre: server umount lustre-OST0000 complete [10541.543251] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10541.552834] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10541.570710] Lustre: Skipped 1 previous similar message [10547.494262] Lustre: server umount lustre-MDT0000 complete [10548.193479] LustreError: 379823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10548.217672] LustreError: 379823:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10551.096180] LustreError: 379803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841356 with bad export cookie 13306653632527496127 [10551.103973] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10551.108960] LustreError: 379803:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10557.485766] Lustre: server umount lustre-MDT0001 complete [10566.067980] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [10568.248227] Key type lgssc unregistered [10568.529668] LNet: 384646:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10568.534546] LNetError: 384646:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10568.552635] LNet: Removed LNI 192.168.206.155@tcp [10569.309809] Key type .llcrypt unregistered [10569.312908] Key type ._llcrypt unregistered [10614.967379] Key type ._llcrypt registered [10614.972773] Key type .llcrypt registered [10615.044163] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10616.062242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10616.118070] alg: No test for adler32 (adler32-zlib) [10617.110391] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [10617.295277] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [10618.951180] Key type lgssc registered [10619.978586] Lustre: Echo OBD driver; http://www.lustre.org/ [10626.886648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10626.900476] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10628.140968] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10628.158660] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10628.273135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10632.532156] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10641.352831] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10641.431433] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10641.462570] Lustre: 387447:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10641.800594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10646.401365] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10653.163786] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10658.490483] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10662.379777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10662.391677] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10662.412721] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10665.445855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10665.455606] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10667.040333] Lustre: server umount lustre-MDT0000 complete [10670.409042] LustreError: 387437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841476 with bad export cookie 1793902766341011438 [10670.416036] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10670.429743] LustreError: 387437:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10670.723464] Lustre: server umount lustre-MDT0001 complete [10679.708747] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 10:38:04 (1787841484) [10685.689233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10686.068315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10690.116135] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10691.552693] LustreError: 389962:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10691.566508] LustreError: 389962:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10696.674145] LustreError: 389961:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10698.734681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10699.117624] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10703.712743] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10710.491974] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10715.998250] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10723.161206] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10723.397288] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10723.700685] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10732.213330] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10740.171936] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10748.770587] Lustre: Permanently deactivating lustre-OST0000 [10748.800528] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10758.656833] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10760.875300] LustreError: 392998:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10760.886935] LustreError: 392998:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10760.898791] LustreError: 392998:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10764.307435] LustreError: 393048:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10764.312610] LustreError: 393048:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10772.620879] Lustre: server umount lustre-OST0000 complete [10775.142959] LustreError: 393297:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10775.151885] LustreError: 393297:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10779.107912] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10779.119330] Lustre: Skipped 1 previous similar message [10779.122542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10779.131433] Lustre: Skipped 2 previous similar messages [10784.247286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10784.258610] Lustre: Skipped 1 previous similar message [10789.345901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10789.355586] Lustre: Skipped 1 previous similar message [10791.391495] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10791.577450] Lustre: server umount lustre-MDT0000 complete [10794.476432] LustreError: 390687:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10794.492356] LustreError: 390687:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10795.211254] LustreError: 389941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841601 with bad export cookie 1793902766341012019 [10795.218580] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10795.220494] LustreError: 389941:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10805.897691] Lustre: server umount lustre-MDT0001 complete [10815.084180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10819.631347] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10822.225884] LustreError: 394456:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10827.858882] Lustre: 394796:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.155@tcp [10829.856413] Lustre: server umount MGS complete [10841.641350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10842.163188] LustreError: 395521:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10842.193295] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10842.200100] Lustre: Skipped 1 previous similar message [10842.250475] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10846.084768] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10847.715457] LustreError: 395522:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10853.761526] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10854.069701] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10857.983592] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10864.897914] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10871.005740] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10872.883172] Lustre: Permanently reactivating lustre-OST0000 [10872.896940] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10872.904327] Lustre: Skipped 2 previous similar messages [10901.983161] LustreError: 397347:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10908.869702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10909.109056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10909.112796] Lustre: Skipped 1 previous similar message [10914.156534] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [10923.137564] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10930.562731] Lustre: server umount lustre-OST0000 complete [10936.289373] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10936.304095] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10936.321936] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10944.068507] Lustre: server umount lustre-MDT0000 complete [10947.047462] LustreError: 396248:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10947.079099] LustreError: 396248:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10948.066919] Lustre: 386769:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787841737/real 1787841737] req@00000000b827edb0 x1874687608062976/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787841753 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10948.073794] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10948.085507] Lustre: 386769:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [10948.104680] Lustre: Skipped 1 previous similar message [10948.118363] LustreError: 395501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841754 with bad export cookie 1793902766341013069 [10948.129067] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10948.130584] LustreError: 395501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10948.479377] Lustre: server umount lustre-MDT0001 complete [10959.070955] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [10961.918932] Key type lgssc unregistered [10962.264629] LNet: 399983:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10962.276122] LNetError: 399983:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10962.308577] LNet: Removed LNI 192.168.206.155@tcp [10963.178550] Key type .llcrypt unregistered [10963.184624] Key type ._llcrypt unregistered [10981.749503] Key type ._llcrypt registered [10981.751704] Key type .llcrypt registered [10981.835666] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [10997.134636] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [10998.531152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10998.543232] alg: No test for adler32 (adler32-zlib) [10999.626391] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [10999.949443] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11001.607159] Key type lgssc registered [11002.677740] Lustre: Echo OBD driver; http://www.lustre.org/ [11038.676577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11038.699803] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11040.007364] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11040.055176] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11040.184919] Lustre: lustre-MDT0000: new disk, initializing [11040.346570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11040.384790] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11045.375502] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11057.642255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11057.782461] Lustre: 403157:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11057.866887] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11057.871311] Lustre: Skipped 1 previous similar message [11057.984558] Lustre: lustre-MDT0001: new disk, initializing [11058.128040] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11058.196048] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11058.204800] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11063.140956] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11075.534359] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11081.184270] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11086.079867] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11086.252552] Lustre: lustre-OST0000: new disk, initializing [11086.255641] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11086.262098] Lustre: 405127:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11086.381287] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11091.184431] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11092.012690] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11092.020482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11092.124245] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11098.457862] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11102.432054] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11102.585168] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11106.843083] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11107.083661] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11112.417899] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11112.423222] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11112.426445] LustreError: Skipped 1 previous similar message [11112.442040] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11117.540945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11117.545178] Lustre: Skipped 2 previous similar messages [11122.661537] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11122.668226] Lustre: Skipped 1 previous similar message [11123.167321] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11123.309610] Lustre: server umount lustre-OST0000 complete [11127.776814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11127.777365] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11127.792309] Lustre: Skipped 2 previous similar messages [11132.980138] Lustre: server umount lustre-MDT0000 complete [11136.352733] LustreError: 403150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787841942 with bad export cookie 1870294868997721682 [11136.357236] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11136.369913] LustreError: 403150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11136.698260] Lustre: server umount lustre-MDT0001 complete [11144.856188] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [11146.774594] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 10:45:50 (1787841950) [11152.150133] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11152.665861] LustreError: 407586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11152.681807] LustreError: 407586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11152.735606] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11156.616944] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11157.992737] LustreError: 407587:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11163.105701] LustreError: 407586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11166.020539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11166.394908] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11171.062916] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11179.551925] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11186.197889] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11193.995451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11194.665214] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11201.410685] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11209.465516] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11211.698482] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [11214.465981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [11219.430826] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11219.439392] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11219.455367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11219.457798] Lustre: Skipped 1 previous similar message [11221.481983] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11221.493236] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11225.689773] Lustre: server umount lustre-OST0000 complete [11231.716719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11231.718634] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11231.733135] Lustre: Skipped 1 previous similar message [11231.752025] Lustre: Skipped 3 previous similar messages [11235.411750] Lustre: server umount lustre-MDT0000 complete [11236.835834] LustreError: 407586:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11236.857121] LustreError: 407586:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11239.323340] LustreError: 407567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842045 with bad export cookie 1870294868997722571 [11239.334384] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11239.337188] LustreError: 407567:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11239.625915] Lustre: server umount lustre-MDT0001 complete [11251.438766] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11254.468880] Key type lgssc unregistered [11254.816650] LNet: 411315:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11254.825800] LNetError: 411315:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11254.846328] LNet: Removed LNI 192.168.206.155@tcp [11255.629782] Key type .llcrypt unregistered [11255.632491] Key type ._llcrypt unregistered [11277.919867] Key type ._llcrypt registered [11277.922181] Key type .llcrypt registered [11278.028592] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11278.291347] Key type .llcrypt unregistered [11278.293506] Key type ._llcrypt unregistered [11295.993625] Key type ._llcrypt registered [11296.000326] Key type .llcrypt registered [11296.133602] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11297.276094] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11297.303398] alg: No test for adler32 (adler32-zlib) [11298.490677] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [11298.741057] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11300.399202] Key type lgssc registered [11301.529374] Lustre: Echo OBD driver; http://www.lustre.org/ [11309.192313] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11309.230057] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11310.740809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11314.839183] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11320.296641] LustreError: 413532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11320.320378] LustreError: 413532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11322.676833] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11322.890396] LustreError: 413533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11322.906546] LustreError: 413533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11322.922321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11327.969298] LustreError: 413532:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11327.984979] LustreError: 413532:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11328.558366] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11333.091043] LustreError: 413527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11333.114431] LustreError: 413527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11334.786727] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11335.159455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11340.273666] LustreError: 413527:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11340.296162] LustreError: 413527:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11341.015296] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11341.433625] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11345.628937] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11355.627130] Lustre: server umount lustre-OST0000 complete [11360.738884] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [11360.744721] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11360.758287] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11362.274927] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11362.278310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11362.288660] Lustre: Skipped 1 previous similar message [11367.393549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11367.404952] Lustre: Skipped 1 previous similar message [11369.547941] Lustre: server umount lustre-MDT0000 complete [11372.448122] Lustre: 412845:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787842162/real 1787842162] req@00000000c79b0321 x1874688322069888/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787842178 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11372.474255] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11372.490359] LustreError: 413533:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11372.503839] LustreError: 413533:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11372.945826] LustreError: 413513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842178 with bad export cookie 7105969441545542852 [11372.954361] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11372.955490] LustreError: 413513:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11373.255516] Lustre: server umount lustre-MDT0001 complete [11384.661718] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11387.481471] Key type lgssc unregistered [11387.865470] LNet: 416682:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11387.888178] LNetError: 416682:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11387.915291] LNet: Removed LNI 192.168.206.155@tcp [11388.773056] Key type .llcrypt unregistered [11388.776079] Key type ._llcrypt unregistered [11412.094466] Key type ._llcrypt registered [11412.096183] Key type .llcrypt registered [11412.254784] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11413.359790] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11413.389406] alg: No test for adler32 (adler32-zlib) [11414.536296] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [11414.792754] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11416.543547] Key type lgssc registered [11417.566122] Lustre: Echo OBD driver; http://www.lustre.org/ [11424.611947] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11424.627601] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11440.159440] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [11445.279271] LustreError: 418075:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.155@tcp: failed processing log, type 4: rc = -110 [11471.968109] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11477.773949] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11486.057372] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11492.354741] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11492.926237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11493.013897] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11497.043177] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11497.958520] LustreError: 418100:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11497.977597] LustreError: 418100:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11503.075356] LustreError: 418101:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11504.499958] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11504.882936] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11509.289393] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11516.864752] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11522.597110] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11530.728884] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11530.744813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11530.762126] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11532.529310] Lustre: server umount lustre-OST0000 complete [11537.891329] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11537.896395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11537.925040] Lustre: Skipped 1 previous similar message [11537.942472] Lustre: Skipped 1 previous similar message [11543.009137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11543.013113] Lustre: Skipped 1 previous similar message [11547.109867] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11547.124088] Lustre: Skipped 1 previous similar message [11547.180711] Lustre: server umount lustre-MDT0000 complete [11550.663250] LustreError: 418082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842356 with bad export cookie 14238799441188731873 [11550.667508] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11550.670799] LustreError: 418082:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11550.959466] Lustre: server umount lustre-MDT0001 complete [11560.518122] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11562.852847] Key type lgssc unregistered [11563.135302] LNet: 421647:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11563.143925] LNetError: 421647:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11563.160820] LNet: Removed LNI 192.168.206.155@tcp [11563.820462] Key type .llcrypt unregistered [11563.827992] Key type ._llcrypt unregistered [11586.451400] Key type ._llcrypt registered [11586.455778] Key type .llcrypt registered [11586.566588] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11587.726925] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11587.763180] alg: No test for adler32 (adler32-zlib) [11588.785792] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [11588.965130] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11590.607563] Key type lgssc registered [11591.929184] Lustre: Echo OBD driver; http://www.lustre.org/ [11599.600483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11599.621822] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11601.033630] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11605.497548] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11610.599498] LustreError: 423064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11610.609752] LustreError: 423064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11613.044661] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11613.518797] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11617.202993] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11624.542612] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11630.526715] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11638.252472] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11638.823124] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11645.287956] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11645.960838] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11653.015967] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11658.726828] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11658.736792] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11658.754265] Lustre: Skipped 2 previous similar messages [11658.780752] Lustre: Skipped 2 previous similar messages [11660.006546] Lustre: server umount lustre-MDT0000 complete [11661.001164] LustreError: 423059:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.55@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11661.896363] LustreError: 423045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.55@tcp arrived at 1787842467 with bad export cookie 18344408317804120649 [11661.918481] LustreError: 423045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11663.759743] LustreError: 423800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787842469 with bad export cookie 18344408317804119760 [11663.775034] LustreError: 423800:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11663.775276] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11663.841277] LustreError: 423064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11663.844019] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11663.857800] LustreError: 423064:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11663.865355] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11666.074214] Lustre: server umount lustre-OST0000 complete [11668.966451] LustreError: 423792:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11668.995673] LustreError: 423792:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11673.128195] LustreError: 423794:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000265cd408 x1874688626161792/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11673.479162] Lustre: server umount lustre-MDT0001 complete [11688.088905] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11691.228752] Key type lgssc unregistered [11691.550357] LNet: 426751:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11691.559326] LNetError: 426751:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11691.579797] LNet: Removed LNI 192.168.206.155@tcp [11692.769485] Key type .llcrypt unregistered [11692.771812] Key type ._llcrypt unregistered [11720.021753] Key type ._llcrypt registered [11720.024394] Key type .llcrypt registered [11720.141944] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11721.351502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11721.419216] alg: No test for adler32 (adler32-zlib) [11722.584512] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [11722.928413] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11724.679211] Key type lgssc registered [11726.011117] Lustre: Echo OBD driver; http://www.lustre.org/ [11733.978576] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11734.004387] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11735.490381] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11740.097328] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11745.765781] LustreError: 428168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11745.787897] LustreError: 428168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11749.359353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11749.937520] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11755.062798] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11766.114133] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11773.814841] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11781.539574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11782.127865] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11788.889180] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11796.448584] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11806.715218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11811.296218] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11811.301425] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11811.312209] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11811.809564] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11811.822903] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11816.324475] Lustre: server umount lustre-MDT0001 complete [11816.929620] LustreError: 428169:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11816.954985] LustreError: 428169:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11822.065266] LustreError: 428168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11822.080636] LustreError: 428168:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11827.169957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11827.172198] LustreError: 428897:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11827.182409] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11827.185968] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11827.227464] Lustre: Skipped 1 previous similar message [11829.458649] Lustre: server umount lustre-OST0000 complete [11832.293441] LustreError: 428168:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11833.639345] Lustre: server umount lustre-MDT0000 complete [11846.976383] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [11850.706264] Key type lgssc unregistered [11851.079844] LNet: 431879:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11851.096251] LNetError: 431879:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11851.118284] LNet: Removed LNI 192.168.206.155@tcp [11852.247524] Key type .llcrypt unregistered [11852.250187] Key type ._llcrypt unregistered [11878.966867] Key type ._llcrypt registered [11878.975766] Key type .llcrypt registered [11879.105784] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [11896.910149] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11898.212858] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11898.378268] alg: No test for adler32 (adler32-zlib) [11899.407390] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [11899.660155] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [11901.359139] Key type lgssc registered [11902.589603] Lustre: Echo OBD driver; http://www.lustre.org/ [11943.783371] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [11959.298138] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11959.352724] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11960.660641] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11960.776841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11960.917715] Lustre: lustre-MDT0000: new disk, initializing [11961.044424] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11961.078698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11966.320674] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11982.588071] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11982.824716] Lustre: lustre-OST0000: new disk, initializing [11982.833758] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11982.842358] Lustre: Skipped 1 previous similar message [11982.847572] Lustre: 436914:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11982.934449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11988.909785] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [11992.678532] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11992.690912] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11992.722447] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11998.341430] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12012.125163] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12012.194689] Lustre: 435897:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12012.242184] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12012.283614] Lustre: lustre-MDT0001: new disk, initializing [12012.351991] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12012.401726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [12012.409099] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [12016.448931] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12026.220174] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [12026.471299] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [12031.865320] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [12032.178955] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [12038.375712] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.55@tcp (stopping) [12040.679230] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [12040.680452] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12040.705469] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [12040.713840] Lustre: Skipped 1 previous similar message [12043.458326] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.55@tcp (stopping) [12043.465616] Lustre: Skipped 1 previous similar message [12043.981918] Lustre: server umount lustre-MDT0001 complete [12050.917266] LustreError: 435905:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12050.951461] LustreError: 435905:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [12056.037920] LustreError: 435908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12056.081399] LustreError: 435908:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12061.164819] LustreError: 435909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12061.190277] LustreError: 435909:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12064.230133] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12064.240366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12064.270432] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12066.273059] LustreError: 435908:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12068.827468] Lustre: server umount lustre-OST0000 complete [12071.397824] LustreError: 435909:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12079.356612] Lustre: server umount lustre-MDT0000 complete [12092.466661] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12095.219112] Key type lgssc unregistered [12095.503704] LNet: 440676:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12095.511548] LNetError: 440676:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12095.533974] LNet: Removed LNI 192.168.206.155@tcp [12096.540511] Key type .llcrypt unregistered [12096.545605] Key type ._llcrypt unregistered [12131.502315] Key type ._llcrypt registered [12131.503825] Key type .llcrypt registered [12131.587699] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [12131.839304] Key type .llcrypt unregistered [12131.842737] Key type ._llcrypt unregistered [12147.139583] Key type ._llcrypt registered [12147.142521] Key type .llcrypt registered [12147.253530] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [12148.119959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12148.263597] alg: No test for adler32 (adler32-zlib) [12149.534741] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [12149.802796] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [12151.455553] Key type lgssc registered [12152.710130] Lustre: Echo OBD driver; http://www.lustre.org/ [12179.077110] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12179.117811] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12180.328515] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12180.367753] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12180.460780] Lustre: lustre-MDT0000: new disk, initializing [12180.506949] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12180.520099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12184.565744] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12195.064502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12195.192794] Lustre: 444049:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12195.240618] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12195.244339] Lustre: Skipped 1 previous similar message [12195.302248] Lustre: lustre-MDT0001: new disk, initializing [12195.397227] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12195.448727] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12195.459491] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12200.265488] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12210.178333] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12216.175871] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12222.596116] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12222.879820] Lustre: lustre-OST0000: new disk, initializing [12222.883866] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12222.892367] Lustre: 446018:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12222.993528] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12229.383711] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12229.710308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12229.719951] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12229.811276] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12238.818769] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12245.474711] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12245.492960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12245.495245] Lustre: Skipped 1 previous similar message [12245.509976] Lustre: Skipped 2 previous similar messages [12250.596116] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12250.605226] Lustre: Skipped 2 previous similar messages [12250.993705] Lustre: server umount lustre-MDT0000 complete [12254.569863] LustreError: 444853:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843060 with bad export cookie 205978398582148779 [12254.587260] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12255.046845] Lustre: server umount lustre-MDT0001 complete [12269.210901] Lustre: server umount lustre-OST0000 complete [12278.260846] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 11:04:42 (1787843082) [12288.309398] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12288.775518] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12293.493826] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12294.123792] LustreError: 448047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12294.141156] LustreError: 448047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12298.211865] LustreError: 448046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12301.355152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12301.622591] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12306.089222] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12313.242367] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12318.937401] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12338.856072] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12339.084919] Lustre: Found index 0 for lustre-OST0000, updating log [12339.267597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12344.635350] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [12351.023652] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12359.729556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [12366.819823] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12366.825689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12366.831647] Lustre: Skipped 1 previous similar message [12366.835220] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12371.730659] Lustre: server umount lustre-OST0000 complete [12388.832023] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12388.839051] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12388.847796] Lustre: Skipped 1 previous similar message [12388.850701] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12388.853010] Lustre: Skipped 3 previous similar messages [12392.843459] Lustre: server umount lustre-MDT0000 complete [12393.440726] LustreError: 448047:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12393.460308] LustreError: 448047:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12395.710637] LustreError: 448026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843201 with bad export cookie 205978398582150172 [12395.712704] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12395.720140] LustreError: 448026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [12395.997167] Lustre: server umount lustre-MDT0001 complete [12399.712440] LNet: 1 local NIs in recovery (showing 1): 192.168.206.155@tcp [12403.516874] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 11:06:48 (1787843208) [13017.571904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13018.134492] LustreError: 454477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13018.152339] LustreError: 454477:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13018.235750] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13022.969132] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13023.712628] LustreError: 454478:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13028.851264] LustreError: 454477:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13031.020074] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13031.386965] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13035.764362] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13042.531852] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13047.529509] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13067.626831] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13067.798177] Lustre: Found index 0 for lustre-OST0000, updating log [13068.035499] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13073.692805] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13077.602151] LNet: 1 local NIs in recovery (showing 1): 192.168.206.155@tcp [13080.313246] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13082.663125] Lustre: server umount lustre-OST0000 complete [13086.274303] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13086.287337] Lustre: Skipped 2 previous similar messages [13086.287907] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13086.303939] Lustre: Skipped 3 previous similar messages [13087.330268] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13087.342710] Lustre: Skipped 1 previous similar message [13088.354553] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13088.359720] Lustre: Skipped 1 previous similar message [13090.404353] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13090.417257] Lustre: Skipped 3 previous similar messages [13092.692187] Lustre: server umount lustre-MDT0000 complete [13093.474759] LustreError: 455718:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13093.502853] LustreError: 455718:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13096.524952] LustreError: 454460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787843902 with bad export cookie 205978398582151257 [13096.536187] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13096.542068] LustreError: 454460:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13096.794429] Lustre: server umount lustre-MDT0001 complete [13126.762962] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [13138.519309] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13171.370223] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 11:19:35 (1787843975) [13190.857812] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [13193.834115] Key type lgssc unregistered [13194.164321] LNet: 461094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13194.179248] LNetError: 461094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13322.216768] LNet: Removed LNI 192.168.206.155@tcp [13323.095199] Key type .llcrypt unregistered [13323.097336] Key type ._llcrypt unregistered [13338.237575] Key type ._llcrypt registered [13338.240684] Key type .llcrypt registered [13338.351545] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13339.499149] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13339.522543] alg: No test for adler32 (adler32-zlib) [13340.566725] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [13340.779059] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [13342.431440] Key type lgssc registered [13343.676977] Lustre: Echo OBD driver; http://www.lustre.org/ [13348.273034] LNet: Added LNI 192.168.206.155@tcp100 [8/256/0/180] [13349.037680] LNet: Added LNI 192.168.206.155@tcp101 [8/256/0/180] [13349.883397] LNet: Added LNI 192.168.206.155@tcp102 [8/256/0/180] [13350.666482] LNet: Added LNI 192.168.206.155@tcp103 [8/256/0/180] [13351.335992] LNet: Added LNI 192.168.206.155@tcp104 [8/256/0/180] [13352.137626] LNet: Added LNI 192.168.206.155@tcp105 [8/256/0/180] [13352.901412] LNet: Added LNI 192.168.206.155@tcp106 [8/256/0/180] [13353.762741] LNet: Added LNI 192.168.206.155@tcp107 [8/256/0/180] [13354.601169] LNet: Added LNI 192.168.206.155@tcp108 [8/256/0/180] [13355.510673] LNet: Added LNI 192.168.206.155@tcp109 [8/256/0/180] [13356.340826] LNet: Added LNI 192.168.206.155@tcp110 [8/256/0/180] [13357.202133] LNet: Added LNI 192.168.206.155@tcp111 [8/256/0/180] [13358.018636] LNet: Added LNI 192.168.206.155@tcp112 [8/256/0/180] [13358.741841] LNet: Added LNI 192.168.206.155@tcp113 [8/256/0/180] [13359.653977] LNet: Added LNI 192.168.206.155@tcp114 [8/256/0/180] [13360.372670] LNet: Added LNI 192.168.206.155@tcp115 [8/256/0/180] [13361.151281] LNet: Added LNI 192.168.206.155@tcp116 [8/256/0/180] [13361.986976] LNet: Added LNI 192.168.206.155@tcp117 [8/256/0/180] [13362.804840] LNet: Added LNI 192.168.206.155@tcp118 [8/256/0/180] [13363.558201] LNet: Added LNI 192.168.206.155@tcp119 [8/256/0/180] [13364.363733] LNet: Added LNI 192.168.206.155@tcp120 [8/256/0/180] [13365.371442] LNet: Added LNI 192.168.206.155@tcp121 [8/256/0/180] [13366.145345] LNet: Added LNI 192.168.206.155@tcp122 [8/256/0/180] [13366.951747] LNet: Added LNI 192.168.206.155@tcp123 [8/256/0/180] [13367.684563] LNet: Added LNI 192.168.206.155@tcp124 [8/256/0/180] [13368.438167] LNet: Added LNI 192.168.206.155@tcp125 [8/256/0/180] [13369.179372] LNet: Added LNI 192.168.206.155@tcp126 [8/256/0/180] [13369.905793] LNet: Added LNI 192.168.206.155@tcp127 [8/256/0/180] [13370.627895] LNet: Added LNI 192.168.206.155@tcp128 [8/256/0/180] [13371.472395] LNet: Added LNI 192.168.206.155@tcp129 [8/256/0/180] [13372.207541] LNet: Added LNI 192.168.206.155@tcp130 [8/256/0/180] [13372.935433] LNet: Added LNI 192.168.206.155@tcp131 [8/256/0/180] [13373.683737] LNet: Added LNI 192.168.206.155@tcp132 [8/256/0/180] [13374.424638] LNet: Added LNI 192.168.206.155@tcp133 [8/256/0/180] [13375.165253] LNet: Added LNI 192.168.206.155@tcp134 [8/256/0/180] [13375.846322] LNet: Added LNI 192.168.206.155@tcp135 [8/256/0/180] [13382.211990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13382.258068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13383.396659] Lustre: 464121:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13383.514294] LNetError: 464149:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13383.528227] LNetError: Protocol error connecting to 192.168.206.155@tcp119 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13383.547069] LNetError: Skipped 7 previous similar messages [13383.673851] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13383.704896] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13383.775309] LNet: 5 local NIs in recovery (showing 5): 192.168.206.155@tcp124, 192.168.206.155@tcp126, 192.168.206.155@tcp127, 192.168.206.155@tcp128, 192.168.206.155@tcp129 [13383.801458] Lustre: lustre-MDT0000: new disk, initializing [13383.868531] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13383.887766] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13384.547013] LNetError: 461577:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13384.561781] LNetError: 461577:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 8 previous similar messages [13384.567038] LNetError: Protocol error connecting to 192.168.206.155@tcp103 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13384.579222] LNetError: Skipped 1 previous similar message [13386.849243] LNetError: 464149:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13386.860324] LNetError: 464149:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [13386.872780] LNetError: Protocol error connecting to 192.168.206.155@tcp124 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13386.895551] LNetError: Skipped 5 previous similar messages [13387.504751] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13390.946610] LNetError: 464145:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13390.957716] LNetError: Protocol error connecting to 192.168.206.155@tcp124 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13394.915980] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13395.108349] Lustre: 464139:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13395.177585] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13395.179961] Lustre: Skipped 1 previous similar message [13395.249438] Lustre: lustre-MDT0001: new disk, initializing [13395.288502] Lustre: 464940:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13395.298918] Lustre: 464940:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13395.455691] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13395.498963] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13395.507022] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13396.755819] Lustre: 465024:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13396.762856] Lustre: 465024:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13398.112884] LNetError: 461577:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13398.129548] LNetError: Protocol error connecting to 192.168.206.155@tcp124 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13399.748024] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13408.721124] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13413.006363] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13413.480855] LNetError: 464147:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13413.491928] LNetError: Protocol error connecting to 192.168.206.155@tcp123 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13418.807518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13419.093565] Lustre: lustre-OST0000: new disk, initializing [13419.097419] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13419.107723] Lustre: 466111:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13419.227724] Lustre: 466116:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13419.237798] Lustre: 466116:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [13419.282026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13424.665038] Lustre: 466410:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.155@tcp: can't add NID 192.168.206.155@tcp130: rc = -75 [13424.677389] Lustre: 466410:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [13424.756837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13424.766702] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13424.895869] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13425.451772] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13430.882191] LNetError: 464149:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13430.900659] LNetError: 464149:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [13430.918664] LNetError: Protocol error connecting to 192.168.206.155@tcp103 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13430.939385] LNetError: Skipped 7 previous similar messages [13434.510187] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13443.167387] LNet: 1 local NIs in recovery (showing 1): 192.168.206.155@tcp124 [13445.605032] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13445.610938] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13445.620428] Lustre: Skipped 1 previous similar message [13445.622149] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13445.629931] Lustre: Skipped 1 previous similar message [13446.783473] Lustre: server umount lustre-OST0000 complete [13450.723920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13450.726223] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13450.730223] Lustre: Skipped 1 previous similar message [13455.847264] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13455.850821] Lustre: Skipped 2 previous similar messages [13456.275911] Lustre: server umount lustre-MDT0000 complete [13459.805165] LustreError: 464939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844265 with bad export cookie 4248595256187022480 [13459.807060] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13459.817967] LustreError: 464939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13460.168192] Lustre: server umount lustre-MDT0001 complete [13468.771446] LNetError: 461577:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.206.155 [13468.787069] LNetError: 461577:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [13468.795201] LNetError: Protocol error connecting to 192.168.206.155@tcp103 at host 192.168.206.155:988: is it running a compatible version of Lustre? [13468.803786] LNetError: Skipped 12 previous similar messages [13469.299408] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [13471.592377] Key type lgssc unregistered [13471.928353] LNet: 468026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13471.936545] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13471.941628] LNet: Removed LNI 192.168.206.155@tcp [13472.992274] LNet: Removed LNI 192.168.206.155@tcp100 [13472.994487] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13473.000409] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13473.003825] LNet: Removed LNI 192.168.206.155@tcp101 [13474.017991] LNet: Removed LNI 192.168.206.155@tcp102 [13474.024348] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13474.031959] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [13474.039519] LNet: Removed LNI 192.168.206.155@tcp103 [13475.103162] LNet: Removed LNI 192.168.206.155@tcp104 [13476.127230] LNet: Removed LNI 192.168.206.155@tcp105 [13476.131817] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13476.155535] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [13476.168152] LNet: Removed LNI 192.168.206.155@tcp106 [13477.215147] LNet: Removed LNI 192.168.206.155@tcp107 [13478.239130] LNet: Removed LNI 192.168.206.155@tcp108 [13479.273476] LNet: Removed LNI 192.168.206.155@tcp109 [13480.287471] LNet: Removed LNI 192.168.206.155@tcp110 [13480.293531] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13480.296652] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [13480.303543] LNet: Removed LNI 192.168.206.155@tcp111 [13481.311275] LNet: Removed LNI 192.168.206.155@tcp112 [13482.335109] LNet: Removed LNI 192.168.206.155@tcp113 [13483.359143] LNet: Removed LNI 192.168.206.155@tcp114 [13484.384180] LNet: Removed LNI 192.168.206.155@tcp115 [13485.407363] LNet: Removed LNI 192.168.206.155@tcp116 [13486.431978] LNet: Removed LNI 192.168.206.155@tcp117 [13487.455135] LNet: Removed LNI 192.168.206.155@tcp118 [13488.479214] LNet: Removed LNI 192.168.206.155@tcp119 [13488.482434] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13488.503052] LNetError: 468026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [13488.525291] LNet: Removed LNI 192.168.206.155@tcp120 [13489.567277] LNet: Removed LNI 192.168.206.155@tcp121 [13490.591143] LNet: Removed LNI 192.168.206.155@tcp122 [13491.615161] LNet: Removed LNI 192.168.206.155@tcp123 [13491.617547] LNet: Removed LNI 192.168.206.155@tcp124 [13492.639273] LNet: Removed LNI 192.168.206.155@tcp125 [13493.663253] LNet: Removed LNI 192.168.206.155@tcp126 [13494.752489] LNet: Removed LNI 192.168.206.155@tcp127 [13495.775169] LNet: Removed LNI 192.168.206.155@tcp128 [13496.799373] LNet: Removed LNI 192.168.206.155@tcp129 [13496.803418] LNet: Removed LNI 192.168.206.155@tcp130 [13496.805403] LNet: Removed LNI 192.168.206.155@tcp131 [13496.807604] LNet: Removed LNI 192.168.206.155@tcp132 [13496.810816] LNet: Removed LNI 192.168.206.155@tcp133 [13496.813601] LNet: Removed LNI 192.168.206.155@tcp134 [13496.823252] LNet: Removed LNI 192.168.206.155@tcp135 [13497.405044] Key type .llcrypt unregistered [13497.407150] Key type ._llcrypt unregistered [13511.126545] Key type ._llcrypt registered [13511.128902] Key type .llcrypt registered [13511.225492] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [13523.636953] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13524.350630] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13524.414150] alg: No test for adler32 (adler32-zlib) [13525.446049] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [13525.676924] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [13527.327179] Key type lgssc registered [13528.306595] Lustre: Echo OBD driver; http://www.lustre.org/ [13556.717950] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 11:26:01 (1787844361) [13569.236811] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [13571.374624] Key type lgssc unregistered [13571.570569] LNet: 472092:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13571.584883] LNetError: 472092:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13571.600954] LNet: Removed LNI 192.168.206.155@tcp [13572.178481] Key type .llcrypt unregistered [13572.180747] Key type ._llcrypt unregistered [13584.603236] Key type ._llcrypt registered [13584.605391] Key type .llcrypt registered [13584.698314] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13585.657208] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13585.681763] alg: No test for adler32 (adler32-zlib) [13586.723860] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [13586.969403] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [13588.687367] Key type lgssc registered [13589.943243] Lustre: Echo OBD driver; http://www.lustre.org/ [13597.780641] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13597.811662] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13599.017741] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13599.047773] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13599.144217] Lustre: lustre-MDT0000: new disk, initializing [13599.209600] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13599.222826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13603.048858] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13612.033399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13612.125365] Lustre: 473283:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13612.164499] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13612.169591] Lustre: Skipped 1 previous similar message [13612.268173] Lustre: lustre-MDT0001: new disk, initializing [13612.370837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13612.412508] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13612.427178] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13615.742224] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13624.127798] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13628.954506] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13634.686824] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13634.909591] Lustre: lustre-OST0000: new disk, initializing [13634.912461] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13634.917669] Lustre: 475249:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13634.990472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13639.631971] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13644.357228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13644.370363] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13644.452666] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13647.354315] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13650.699280] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing lnet_if_list [13654.669753] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [13654.704942] LNet: Added LNI 192.168.206.155@tcp10 [8/256/0/180] [13659.990812] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [13660.021276] LNet: Added LNI 192.168.206.155@tcp11 [8/256/0/180] [13663.236808] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [13663.272635] LNet: Added LNI 192.168.206.155@tcp12 [8/256/0/180] [13666.616160] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [13666.642720] LNet: Added LNI 192.168.206.155@tcp13 [8/256/0/180] [13670.676462] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [13670.697868] LNet: Added LNI 192.168.206.155@tcp14 [8/256/0/180] [13675.027878] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [13675.066808] LNet: Added LNI 192.168.206.155@tcp15 [8/256/0/180] [13679.305181] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [13679.340615] LNet: Added LNI 192.168.206.155@tcp16 [8/256/0/180] [13683.289877] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [13683.324828] LNet: Added LNI 192.168.206.155@tcp17 [8/256/0/180] [13687.013289] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [13687.035206] LNet: Added LNI 192.168.206.155@tcp18 [8/256/0/180] [13690.720284] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [13690.743556] LNet: Added LNI 192.168.206.155@tcp19 [8/256/0/180] [13694.663239] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [13694.712679] LNet: Added LNI 192.168.206.155@tcp20 [8/256/0/180] [13698.788813] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [13698.830110] LNet: Added LNI 192.168.206.155@tcp21 [8/256/0/180] [13703.428482] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [13703.470720] LNet: Added LNI 192.168.206.155@tcp22 [8/256/0/180] [13708.825297] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [13708.875430] LNet: Added LNI 192.168.206.155@tcp23 [8/256/0/180] [13714.540854] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [13714.590879] LNet: Added LNI 192.168.206.155@tcp24 [8/256/0/180] [13720.359206] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [13720.380676] LNet: Added LNI 192.168.206.155@tcp25 [8/256/0/180] [13725.385128] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [13725.417197] LNet: Added LNI 192.168.206.155@tcp26 [8/256/0/180] [13730.702110] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [13730.730444] LNet: Added LNI 192.168.206.155@tcp27 [8/256/0/180] [13735.535790] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [13735.573880] LNet: Added LNI 192.168.206.155@tcp28 [8/256/0/180] [13740.643989] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [13740.666378] LNet: Added LNI 192.168.206.155@tcp29 [8/256/0/180] [13747.061349] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13747.116738] LNet: Added LNI 192.168.206.155@tcp30 [8/256/0/180] [13752.907978] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13752.936537] LNet: Added LNI 192.168.206.155@tcp31 [8/256/0/180] [13758.957694] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13758.981560] LNet: Added LNI 192.168.206.155@tcp32 [8/256/0/180] [13765.102157] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13765.133599] LNet: Added LNI 192.168.206.155@tcp33 [8/256/0/180] [13771.523826] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13771.553440] LNet: Added LNI 192.168.206.155@tcp34 [8/256/0/180] [13778.550151] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13778.573928] LNet: Added LNI 192.168.206.155@tcp35 [8/256/0/180] [13785.093643] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13785.132727] LNet: Added LNI 192.168.206.155@tcp36 [8/256/0/180] [13792.301716] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13792.333725] LNet: Added LNI 192.168.206.155@tcp37 [8/256/0/180] [13799.463576] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13799.492754] LNet: Added LNI 192.168.206.155@tcp38 [8/256/0/180] [13807.372894] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13807.404690] LNet: Added LNI 192.168.206.155@tcp39 [8/256/0/180] [13815.302596] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13815.353728] LNet: Added LNI 192.168.206.155@tcp40 [8/256/0/180] [13822.313636] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13822.345681] LNet: Added LNI 192.168.206.155@tcp41 [8/256/0/180] [13829.685137] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13829.732815] LNet: Added LNI 192.168.206.155@tcp42 [8/256/0/180] [13837.049569] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13837.098395] LNet: Added LNI 192.168.206.155@tcp43 [8/256/0/180] [13844.539486] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13844.570689] LNet: Added LNI 192.168.206.155@tcp44 [8/256/0/180] [13852.063261] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13852.090721] LNet: Added LNI 192.168.206.155@tcp45 [8/256/0/180] [13859.375932] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13859.401414] LNet: Added LNI 192.168.206.155@tcp46 [8/256/0/180] [13866.084915] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13866.124760] LNet: Added LNI 192.168.206.155@tcp47 [8/256/0/180] [13873.281426] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13873.311748] LNet: Added LNI 192.168.206.155@tcp48 [8/256/0/180] [13880.757316] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13880.781231] LNet: Added LNI 192.168.206.155@tcp49 [8/256/0/180] [13888.249174] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13888.269663] LNet: Added LNI 192.168.206.155@tcp50 [8/256/0/180] [13904.353598] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13904.364623] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13904.378326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13905.890794] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13905.895529] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13905.899340] Lustre: Skipped 1 previous similar message [13905.920722] Lustre: Skipped 1 previous similar message [13911.010068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13911.018877] Lustre: Skipped 3 previous similar messages [13916.129101] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13916.133691] Lustre: Skipped 2 previous similar messages [13917.151139] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [13917.405155] Lustre: server umount lustre-MDT0000 complete [13920.557971] LustreError: 473275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844726 with bad export cookie 12867160153185902040 [13920.558639] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13920.569987] LustreError: 473275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13920.844682] Lustre: server umount lustre-MDT0001 complete [13937.632612] Lustre: 472556:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844727/real 1787844727] req@00000000cb081b4e x1874690721344000/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787844743 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13937.665851] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13937.921911] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [13941.729235] Lustre: 472557:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844731/real 1787844731] req@00000000d154d3e6 x1874690721344128/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787844747 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13942.752092] Lustre: 472559:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844732/real 1787844732] req@000000006cb4fcbf x1874690721344256/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787844748 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13945.882989] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13946.911138] Lustre: 472559:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844736/real 1787844736] req@0000000028dc27d7 x1874690721344512/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787844752 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13951.967480] Lustre: 472557:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787844741/real 1787844741] req@000000001be4666f x1874690721344640/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787844757 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13965.595868] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13969.816551] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13971.690635] LustreError: 475242:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13971.693543] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13971.711164] LustreError: 475242:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13976.943730] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13977.057364] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [13977.207565] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13980.711646] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [13982.316012] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13986.365868] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13990.813428] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14006.754965] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14006.766510] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14006.787410] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14007.780824] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14020.063711] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14020.275797] Lustre: server umount lustre-OST0000 complete [14028.260186] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14028.265983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14028.274758] Lustre: Skipped 1 previous similar message [14028.280928] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14028.285118] Lustre: Skipped 7 previous similar messages [14030.207346] Lustre: server umount lustre-MDT0000 complete [14032.939313] LustreError: 473275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787844838 with bad export cookie 12867160153185904665 [14032.942784] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14032.950202] LustreError: 473275:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14033.157131] Lustre: server umount lustre-MDT0001 complete [14040.189802] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [14042.486563] Key type lgssc unregistered [14042.857558] LNet: 502269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14042.866241] LNetError: 502269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14042.874630] LNet: Removed LNI 192.168.206.155@tcp [14042.876298] LNet: Removed LNI 192.168.206.155@tcp10 [14042.894162] LNet: Removed LNI 192.168.206.155@tcp11 [14042.900266] LNet: Removed LNI 192.168.206.155@tcp12 [14042.903507] LNet: Removed LNI 192.168.206.155@tcp13 [14042.909390] LNet: Removed LNI 192.168.206.155@tcp14 [14042.915657] LNet: Removed LNI 192.168.206.155@tcp15 [14042.921185] LNet: Removed LNI 192.168.206.155@tcp16 [14042.926942] LNet: Removed LNI 192.168.206.155@tcp17 [14042.933426] LNet: Removed LNI 192.168.206.155@tcp18 [14042.944266] LNet: Removed LNI 192.168.206.155@tcp19 [14042.950347] LNet: Removed LNI 192.168.206.155@tcp20 [14042.954689] LNet: Removed LNI 192.168.206.155@tcp21 [14042.958467] LNet: Removed LNI 192.168.206.155@tcp22 [14042.961270] LNet: Removed LNI 192.168.206.155@tcp23 [14042.963919] LNet: Removed LNI 192.168.206.155@tcp24 [14042.966476] LNet: Removed LNI 192.168.206.155@tcp25 [14042.968966] LNet: Removed LNI 192.168.206.155@tcp26 [14042.970986] LNet: Removed LNI 192.168.206.155@tcp27 [14042.973924] LNet: Removed LNI 192.168.206.155@tcp28 [14042.976647] LNet: Removed LNI 192.168.206.155@tcp29 [14042.979909] LNet: Removed LNI 192.168.206.155@tcp30 [14042.983480] LNet: Removed LNI 192.168.206.155@tcp31 [14042.985700] LNet: Removed LNI 192.168.206.155@tcp32 [14042.988375] LNet: Removed LNI 192.168.206.155@tcp33 [14042.991051] LNet: Removed LNI 192.168.206.155@tcp34 [14042.993260] LNet: Removed LNI 192.168.206.155@tcp35 [14042.997333] LNet: Removed LNI 192.168.206.155@tcp36 [14043.000425] LNet: Removed LNI 192.168.206.155@tcp37 [14043.002257] LNet: Removed LNI 192.168.206.155@tcp38 [14043.004674] LNet: Removed LNI 192.168.206.155@tcp39 [14043.006841] LNet: Removed LNI 192.168.206.155@tcp40 [14043.009162] LNet: Removed LNI 192.168.206.155@tcp41 [14043.011986] LNet: Removed LNI 192.168.206.155@tcp42 [14043.014378] LNet: Removed LNI 192.168.206.155@tcp43 [14043.017233] LNet: Removed LNI 192.168.206.155@tcp44 [14043.029826] LNet: Removed LNI 192.168.206.155@tcp45 [14043.037864] LNet: Removed LNI 192.168.206.155@tcp46 [14043.049459] LNet: Removed LNI 192.168.206.155@tcp47 [14043.056154] LNet: Removed LNI 192.168.206.155@tcp48 [14043.062269] LNet: Removed LNI 192.168.206.155@tcp49 [14043.081616] LNet: Removed LNI 192.168.206.155@tcp50 [14043.915233] Key type .llcrypt unregistered [14043.924383] Key type ._llcrypt unregistered [14070.264399] Key type ._llcrypt registered [14070.266973] Key type .llcrypt registered [14070.342313] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [14084.718560] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14085.978172] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14086.067786] alg: No test for adler32 (adler32-zlib) [14087.207696] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [14087.503668] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [14089.167623] Key type lgssc registered [14090.274444] Lustre: Echo OBD driver; http://www.lustre.org/ [14118.619036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14118.655482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14119.948317] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14119.970219] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14120.027756] Lustre: lustre-MDT0000: new disk, initializing [14120.075581] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14120.092599] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14124.258335] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14132.912558] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14133.017076] Lustre: 505474:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14133.067661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14133.073096] Lustre: Skipped 1 previous similar message [14133.164927] Lustre: lustre-MDT0001: new disk, initializing [14133.286830] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14133.324415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14133.335933] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14137.153353] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14144.880511] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14148.994486] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14155.434348] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14155.558790] Lustre: *** cfs_fail_loc=90f, val=0*** [14155.627874] Lustre: lustre-OST0000: new disk, initializing [14155.639310] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14155.666281] Lustre: 507490:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14155.746175] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14160.929050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14160.942195] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14161.052878] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14161.053123] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14169.117230] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14176.231490] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14176.234166] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14176.251173] Lustre: Skipped 1 previous similar message [14176.259194] Lustre: Skipped 1 previous similar message [14181.354594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14181.364918] Lustre: Skipped 1 previous similar message [14181.823410] Lustre: server umount lustre-OST0000 complete [14186.465150] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14186.488788] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14186.494782] Lustre: Skipped 1 previous similar message [14191.592373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14191.600585] Lustre: Skipped 1 previous similar message [14191.738674] Lustre: server umount lustre-MDT0000 complete [14195.217973] LustreError: 505468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845001 with bad export cookie 8117424967636359774 [14195.238605] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14195.567858] Lustre: server umount lustre-MDT0001 complete [14203.849853] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [14206.387608] Key type lgssc unregistered [14206.692192] LNet: 509398:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14206.698911] LNetError: 509398:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14207.724189] LNet: Removed LNI 192.168.206.155@tcp [14208.511885] Key type .llcrypt unregistered [14208.514234] Key type ._llcrypt unregistered [14224.329598] Key type ._llcrypt registered [14224.331625] Key type .llcrypt registered [14224.448967] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14225.626267] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14225.650190] alg: No test for adler32 (adler32-zlib) [14226.785472] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [14227.065591] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [14228.767100] Key type lgssc registered [14229.715731] Lustre: Echo OBD driver; http://www.lustre.org/ [14235.397849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14235.414888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14236.739392] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14240.537488] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14246.883281] LustreError: 510648:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14246.895779] LustreError: 510648:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14248.769374] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14249.103803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14253.155965] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14259.987227] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14264.972873] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14271.791229] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14272.224974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14277.783968] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14284.705076] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14295.681251] loop0: detected capacity change from 0 to 204800000 [14295.892496] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14295.899917] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14295.912500] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14301.627059] loop0: detected capacity change from 0 to 204800000 [14301.869084] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14301.891533] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14301.925430] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [14307.609155] loop0: detected capacity change from 0 to 204800000 [14312.774897] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14312.936420] Lustre: fs15246-OST0000: new disk, initializing [14312.940941] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [14312.953662] Lustre: 514239:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14313.023190] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14318.631871] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14327.875410] loop1: detected capacity change from 0 to 204800000 [14333.193707] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14333.297620] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [14333.364707] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [14333.483758] Lustre: fs15246-MDT0000: new disk, initializing [14333.653616] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14333.683050] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14333.838857] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [14334.882854] Lustre: 515363:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000049113a2d x1874691392321536/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787845156 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [14337.807709] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14343.972821] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [14343.984338] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [14344.017109] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [14358.523507] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14358.545264] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [14362.393737] Lustre: server umount fs15246-MDT0000 complete [14363.631021] LustreError: 510643:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14368.757550] LustreError: 510643:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14370.685685] Lustre: server umount fs15246-OST0000 complete [14381.029979] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14381.038105] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14381.050609] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14384.098820] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14384.101109] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14384.103099] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14384.125379] Lustre: Skipped 1 previous similar message [14386.387444] Lustre: server umount lustre-OST0000 complete [14392.288256] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14392.302406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14392.319663] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14399.466966] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14399.479019] Lustre: Skipped 3 previous similar messages [14404.063358] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14404.248123] Lustre: server umount lustre-MDT0000 complete [14404.579437] LustreError: 510644:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14404.595490] LustreError: 510644:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [14407.159612] LustreError: 510627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845213 with bad export cookie 2975948495969380106 [14407.167239] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14407.177685] LustreError: 510627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [14407.513911] Lustre: server umount lustre-MDT0001 complete [14415.553436] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [14417.891598] Key type lgssc unregistered [14418.162557] LNet: 518461:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14418.169077] LNetError: 518461:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14418.185436] LNet: Removed LNI 192.168.206.155@tcp [14418.806853] Key type .llcrypt unregistered [14418.808802] Key type ._llcrypt unregistered [14432.190478] Key type ._llcrypt registered [14432.195023] Key type .llcrypt registered [14432.287967] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14433.325766] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14433.341444] alg: No test for adler32 (adler32-zlib) [14434.434208] Lustre: Lustre: Build Version: 2.17.57_60_g13a1c7b [14434.684912] LNet: Added LNI 192.168.206.155@tcp [8/256/0/180] [14436.447765] Key type lgssc registered [14437.301369] Lustre: Echo OBD driver; http://www.lustre.org/ [14468.722451] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [14478.570842] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14509.724701] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14509.758852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14511.058667] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14511.100443] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14511.231686] Lustre: lustre-MDT0000: new disk, initializing [14511.380704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14511.409630] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14515.977532] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14525.242883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14525.377759] Lustre: 522905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14525.478367] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14525.481857] Lustre: Skipped 1 previous similar message [14525.583431] Lustre: lustre-MDT0001: new disk, initializing [14525.706080] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14525.759927] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14525.776195] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14530.175748] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14538.059290] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14542.258305] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14548.361445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14548.628370] Lustre: lustre-OST0000: new disk, initializing [14548.635714] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14548.640927] Lustre: 524873:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14548.711379] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14554.120281] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_default_debug -1 all [14557.244699] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14557.259402] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14557.346205] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14562.239373] Lustre: DEBUG MARKER: oleg655-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14566.670092] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [14567.017413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [14572.398677] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [14572.784787] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [14578.149699] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14578.158082] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [14578.163142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14578.187966] Lustre: Skipped 1 previous similar message [14581.615458] Lustre: server umount lustre-OST0000 complete [14587.361315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14587.376239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14587.394646] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14587.406657] Lustre: Skipped 1 previous similar message [14588.387904] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14588.422420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14588.441697] Lustre: Skipped 1 previous similar message [14593.185772] Lustre: server umount lustre-MDT0000 complete [14596.444530] LustreError: 522896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787845402 with bad export cookie 2864579989950458026 [14596.457300] LustreError: MGC192.168.206.155@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14596.472054] LustreError: 522896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14597.066737] Lustre: server umount lustre-MDT0001 complete [14621.601589] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing set_hostid [14631.420459] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing load_modules_local [14657.384704] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 14392 sec ========= 11:44:21 (1787845461) [14658.868682] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 11:44:23 (1787845463) === [14661.779452] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 11:44:26 (1787845466) === [14696.688428] Lustre: DEBUG MARKER: oleg655-server.virtnet: executing unload_modules_local [14699.200529] Key type lgssc unregistered [14699.516073] LNet: 531522:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14699.528527] LNetError: 531522:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14699.551678] LNet: Removed LNI 192.168.206.155@tcp [14700.239378] Key type .llcrypt unregistered [14700.243126] Key type ._llcrypt unregistered