[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 522183800 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.002313] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.004010] kvm-guest: setup PV IPIs [ 0.006316] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.007000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.007022] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.008011] pid_max: default: 32768 minimum: 301 [ 0.009000] LSM: Security Framework initializing [ 0.009052] Yama: becoming mindful. [ 0.010030] SELinux: Initializing. [ 0.011067] *** VALIDATE selinux *** [ 0.018052] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.023458] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.024158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.026091] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.027102] *** VALIDATE tmpfs *** [ 0.029351] *** VALIDATE proc *** [ 0.030215] *** VALIDATE cgroup *** [ 0.031009] *** VALIDATE cgroup2 *** [ 0.032264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034326] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035009] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036027] Spectre V2 : User space: Vulnerable [ 0.037015] Speculative Store Bypass: Vulnerable [ 0.040516] debug: unmapping init [mem 0xffffffff84859000-0xffffffff84860fff] [ 0.042153] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043673] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044042] ... version: 2 [ 0.045036] ... bit width: 48 [ 0.046053] ... generic registers: 4 [ 0.047000] ... value mask: 0000ffffffffffff [ 0.047038] ... max period: 00007fffffffffff [ 0.048031] ... fixed-purpose events: 3 [ 0.049029] ... event mask: 000000070000000f [ 0.050294] rcu: Hierarchical SRCU implementation. [ 0.052684] smp: Bringing up secondary CPUs ... [ 0.053725] x86: Booting SMP configuration: [ 0.054055] .... node #0, CPUs: #1 #2 #3 [ 0.067654] smp: Brought up 1 node, 4 CPUs [ 0.068770] smpboot: Max logical packages: 1 [ 0.069011] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.139020] node 0 deferred pages initialised in 66ms [ 0.142009] devtmpfs: initialized [ 0.143233] x86/mm: Memory block size: 128MB [ 0.145798] gcov: version magic: 0x41383552 [ 0.147406] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.151091] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.153316] pinctrl core: initialized pinctrl subsystem [ 0.155201] [ 0.156007] ************************************************************* [ 0.158016] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.161015] ** ** [ 0.163013] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.165013] ** ** [ 0.167015] ** This means that this kernel is built to expose internal ** [ 0.169016] ** IOMMU data structures, which may compromise security on ** [ 0.172014] ** your system. ** [ 0.175012] ** ** [ 0.177008] ** If you see this message and you are not debugging the ** [ 0.178008] ** kernel, report this immediately to your vendor! ** [ 0.181014] ** ** [ 0.183014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.186014] ************************************************************* [ 0.188773] NET: Registered protocol family 16 [ 0.190443] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.193063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.195063] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.199140] cpuidle: using governor menu [ 0.200498] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.202407] PCI: Using configuration type 1 for base access [ 0.204094] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.211091] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.212012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.214116] cryptd: max_cpu_qlen set to 1000 [ 0.215179] ACPI: Added _OSI(Module Device) [ 0.216007] ACPI: Added _OSI(Processor Device) [ 0.217008] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.218009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.222902] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.226408] ACPI: Interpreter enabled [ 0.227048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.227689] ACPI: Using IOAPIC for interrupt routing [ 0.229089] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.231239] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.237701] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.239026] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.241012] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.243060] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.246339] acpiphp: Slot [2] registered [ 0.247135] acpiphp: Slot [5] registered [ 0.249110] acpiphp: Slot [6] registered [ 0.251093] acpiphp: Slot [7] registered [ 0.252128] acpiphp: Slot [8] registered [ 0.254116] acpiphp: Slot [9] registered [ 0.255123] acpiphp: Slot [10] registered [ 0.256090] acpiphp: Slot [3] registered [ 0.257088] acpiphp: Slot [4] registered [ 0.259081] acpiphp: Slot [11] registered [ 0.260098] acpiphp: Slot [12] registered [ 0.262089] acpiphp: Slot [13] registered [ 0.263126] acpiphp: Slot [14] registered [ 0.263856] acpiphp: Slot [15] registered [ 0.265069] acpiphp: Slot [16] registered [ 0.265961] acpiphp: Slot [17] registered [ 0.267073] acpiphp: Slot [18] registered [ 0.268092] acpiphp: Slot [19] registered [ 0.269101] acpiphp: Slot [20] registered [ 0.271087] acpiphp: Slot [21] registered [ 0.272080] acpiphp: Slot [22] registered [ 0.274108] acpiphp: Slot [23] registered [ 0.275073] acpiphp: Slot [24] registered [ 0.275897] acpiphp: Slot [25] registered [ 0.276050] acpiphp: Slot [26] registered [ 0.276887] acpiphp: Slot [27] registered [ 0.278063] acpiphp: Slot [28] registered [ 0.280095] acpiphp: Slot [29] registered [ 0.280956] acpiphp: Slot [30] registered [ 0.282091] acpiphp: Slot [31] registered [ 0.283075] PCI host bridge to bus 0000:00 [ 0.285020] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.287023] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.289026] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.292034] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.294018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.296021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.298145] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.299723] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.302918] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.309518] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.313040] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.315016] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.316014] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.318014] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.319420] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.321577] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.323024] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.324678] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.328982] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.338017] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.341011] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.345796] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.352017] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.357013] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.370017] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.379000] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.387014] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.397018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.419021] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.436287] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.446016] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.454017] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.472071] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.486172] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.495021] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.504019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.524017] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.534976] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.543017] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.553014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.566015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.575018] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.582033] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.590016] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.608020] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.621632] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.624577] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.628496] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.630437] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.633387] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.640042] iommu: Default domain type: Passthrough [ 0.642470] SCSI subsystem initialized [ 0.644157] ACPI: bus type USB registered [ 0.646220] usbcore: registered new interface driver usbfs [ 0.648077] usbcore: registered new interface driver hub [ 0.650065] usbcore: registered new device driver usb [ 0.651163] pps_core: LinuxPPS API ver. 1 registered [ 0.653012] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.656067] PTP clock support registered [ 0.659050] EDAC MC: Ver: 3.0.0 [ 0.660371] PCI: Using ACPI for IRQ routing [ 0.662490] NetLabel: Initializing [ 0.663000] NetLabel: domain hash size = 128 [ 0.665013] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.667080] NetLabel: unlabeled traffic allowed by default [ 0.669134] vgaarb: loaded [ 0.671299] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.673012] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.680181] clocksource: Switched to clocksource kvm-clock [ 0.799702] VFS: Disk quotas dquot_6.6.0 [ 0.801055] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.802960] *** VALIDATE ramfs *** [ 0.804062] *** VALIDATE hugetlbfs *** [ 0.805543] pnp: PnP ACPI init [ 0.808346] pnp: PnP ACPI: found 6 devices [ 0.823315] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.826272] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.828020] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.830028] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.832137] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.834460] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.837093] NET: Registered protocol family 2 [ 0.839081] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.843026] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.845429] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.849157] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.851477] TCP: Hash tables configured (established 65536 bind 65536) [ 0.853104] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.854874] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.857603] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.860558] NET: Registered protocol family 1 [ 0.862880] RPC: Registered named UNIX socket transport module. [ 0.864552] RPC: Registered udp transport module. [ 0.866023] RPC: Registered tcp transport module. [ 0.867730] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.870117] NET: Registered protocol family 44 [ 0.871632] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.873902] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.875904] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.878399] PCI: CLS 0 bytes, default 64 [ 0.880908] Unpacking initramfs... [ 2.281100] debug: unmapping init [mem 0xffff88b37cc54000-0xffff88b37ffbffff] [ 2.285589] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.288050] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.290941] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.750197] Initialise system trusted keyrings [ 2.751956] Key type blacklist registered [ 2.753748] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.762341] zbud: loaded [ 2.765310] *** VALIDATE nfs *** [ 2.766407] *** VALIDATE nfs4 *** [ 2.768096] pstore: using deflate compression [ 2.771859] Platform Keyring initialized [ 2.917965] NET: Registered protocol family 38 [ 2.925243] Key type asymmetric registered [ 2.927320] Asymmetric key parser 'x509' registered [ 2.941220] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.950736] io scheduler mq-deadline registered [ 2.953656] io scheduler kyber registered [ 2.957362] io scheduler bfq registered [ 2.960580] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.963655] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.972307] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.984692] ACPI: Power Button [PWRF] [ 3.001668] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.039615] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.090843] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.107726] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.168410] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.205202] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.243899] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.253418] Non-volatile memory driver v1.3 [ 3.256435] Linux agpgart interface v0.103 [ 3.418525] virtio_blk virtio1: [vda] 146152 512-byte logical blocks (74.8 MB/71.4 MiB) [ 3.431894] vda: detected capacity change from 0 to 74829824 [ 3.467210] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.474564] vdb: detected capacity change from 0 to 1073741824 [ 3.511510] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.522493] vdc: detected capacity change from 0 to 2621440000 [ 3.559401] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.563570] vdd: detected capacity change from 0 to 2621440000 [ 3.599166] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.605554] vde: detected capacity change from 0 to 4294967296 [ 3.651427] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.658443] vdf: detected capacity change from 0 to 4294967296 [ 3.671951] libphy: Fixed MDIO Bus: probed [ 3.715358] usbcore: registered new interface driver usbserial_generic [ 3.719085] usbserial: USB Serial support registered for generic [ 3.720680] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.726899] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.729071] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.732110] mousedev: PS/2 mouse device common for all mice [ 3.735287] rtc_cmos 00:05: RTC can wake from S4 [ 3.738320] rtc_cmos 00:05: registered as rtc0 [ 3.738717] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.739441] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.745653] intel_pstate: CPU model not supported [ 3.750356] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.758505] hid: raw HID events driver (C) Jiri Kosina [ 3.763206] usbcore: registered new interface driver usbhid [ 3.765229] usbhid: USB HID core driver [ 3.766844] drop_monitor: Initializing network drop monitor service [ 3.768557] Initializing XFRM netlink socket [ 3.770724] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.772195] NET: Registered protocol family 10 [ 3.777072] Segment Routing with IPv6 [ 3.786149] NET: Registered protocol family 17 [ 3.790425] mpls_gso: MPLS GSO support [ 3.798398] RAS: Correctable Errors collector initialized. [ 3.800985] AVX version of gcm_enc/dec engaged. [ 3.803077] AES CTR mode by8 optimization enabled [ 3.950657] sched_clock: Marking stable (3950309672, 0)->(4777701413, -827391741) [ 3.955090] registered taskstats version 1 [ 3.962718] Loading compiled-in X.509 certificates [ 3.964969] zswap: loaded using pool lzo/zbud [ 4.035203] Key type big_key registered [ 4.075196] Key type encrypted registered [ 4.076934] ima: No TPM chip found, activating TPM-bypass! [ 4.079299] ima: Allocated hash algorithm: sha1 [ 4.081818] ima: No architecture policies found [ 4.084251] evm: Initialising EVM extended attributes: [ 4.087715] evm: security.selinux [ 4.090179] evm: security.ima [ 4.093637] evm: security.capability [ 4.096551] evm: HMAC attrs: 0x1 [ 4.102118] rtc_cmos 00:05: setting system clock to 2026-08-24 14:49:38 UTC (1787582978) [ 4.115676] debug: unmapping init [mem 0xffffffff85803000-0xffffffff859fffff] [ 4.124119] debug: unmapping init [mem 0xffffffff84582000-0xffffffff84858fff] [ 4.135096] Write protecting the kernel read-only data: 28672k [ 4.140480] debug: unmapping init [mem 0xffffffff82c03000-0xffffffff82dfffff] [ 4.144158] debug: unmapping init [mem 0xffffffff83514000-0xffffffff835fffff] [ 4.227954] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 4.243261] systemd[1]: Detected virtualization kvm. [ 4.247267] systemd[1]: Detected architecture x86-64. [ 4.249568] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.291034] systemd[1]: No hostname configured. [ 4.294644] systemd[1]: Set hostname to . [ 4.297132] random: systemd: uninitialized urandom read (16 bytes read) [ 4.305362] systemd[1]: Initializing machine ID from random generator. [ 4.367342] random: ln: uninitialized urandom read (6 bytes read) [ 4.611488] random: systemd: uninitialized urandom read (16 bytes read) [ 4.615827] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ 4.623585] systemd[1]: Reached target Local File Systems. [ OK ] Reached target Local File Systems. [ 4.633802] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Initrd Root Device. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Slices. [ OK ] Listening on Journal Socket. Starting Journal Service... Starting Create Volatile Files and Directories... Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... [ OK ] Reached target Swap. [ OK ] Reached target Timers. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Local Encrypted Volumes. Starting Apply Kernel Variables... [ OK ] Reached target Sockets. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.336370] device-mapper: uevent: version 1.0.3 [ 6.343711] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 8.062477] virtio_net virtio0 ens2: renamed from eth0 [ 8.205506] random: fast init done [ 8.503102] scsi host0: ata_piix [ 8.519024] scsi host1: ata_piix [ 8.521412] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 8.534906] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.538553] dracut-initqueue[587]: RTNETLINK answers: File exists [ 13.605346] random: crng init done [ 13.609058] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 15.652527] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target System Initialization. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 18.259477] printk: systemd: 26 output lines suppressed due to ratelimiting [ 19.092760] SELinux: Disabled at runtime. [ 19.211813] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 19.230972] systemd[1]: Detected virtualization kvm. [ 19.237676] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 21.023809] systemd[1]: initrd-switch-root.service: Succeeded. [ 21.039420] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 21.062869] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 21.078369] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 21.101277] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 21.114741] systemd[1]: Starting Journal Service... Starting Journal Service... [ 21.140951] systemd[1]: Activating swap /dev/disk/by-label/SWAP... Activating swap /dev/disk/by-label/SWAP... Mounting Kernel Debug File System... Mounting Huge Pages File System... [ 21.187853] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on Process Core Dump Socket. [ OK ] Stopped target Switch Root. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. Mounting POSIX Message Queue File System... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Listening on udev Control Socket. Starting Apply Kernel Variables... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Stopped target Initrd Root File System. [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Kernel Socket. Starting udev Coldplug all Devices... Starting Remount Root and Kernel File Systems... [ OK ] Started Journal Service. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... [ OK ] Reached target Swap. Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 22.625782] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 23.821464] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 23.889733] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 24.538452] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 24.599698] EDAC sbridge: Ver: 1.1.2 [* ] A start job is running for Configur…-only root support (7s / no limit) [** ] A start job is running for Configur…-only root support (8s / no limit) [*** ] A start job is running for Configur…-only root support (8s / no limit)[ 29.861739] Key type dns_resolver registered [ *** ] A start job is running for Configur…-only root support (9s / no limit) [ *** ] A start job is running for Configur…only root support (10s / no limit)[ 31.123215] NFS: Registering the id_resolver key type [ 31.135892] Key type id_resolver registered [ 31.146048] Key type id_legacy registered [ ***] A start job is running for Configur…only root support (10s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Load/Save Random Seed... Starting Create Volatile Files and Directories... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Load/Save Random Seed. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started D-Bus System Message Bus. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started Login Service. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg103-server login: [ 76.657173] hrtimer: interrupt took 4525888 ns [ 92.710398] libcfs: loading out-of-tree module taints kernel. [ 92.791511] Key type ._llcrypt registered [ 92.795996] Key type .llcrypt registered [ 92.939567] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 114.923901] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 116.469293] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 116.485575] alg: No test for adler32 (adler32-zlib) [ 118.165536] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 118.977394] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 120.744287] Key type lgssc registered [ 122.163426] Lustre: Echo OBD driver; http://www.lustre.org/ [ 138.564279] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 177.551098] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 191.676793] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 191.801688] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 193.105781] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 193.132515] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 193.224562] Lustre: lustre-MDT0000: new disk, initializing [ 193.361099] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 193.401791] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 198.773357] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 215.966617] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 216.207681] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 216.327978] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 216.332230] Lustre: Skipped 1 previous similar message [ 216.407501] Lustre: lustre-MDT0001: new disk, initializing [ 216.468250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 216.492222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 216.497741] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 220.359403] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 225.793719] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 239.436445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 239.880474] Lustre: lustre-OST0000: new disk, initializing [ 239.887621] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 239.905350] Lustre: 8457:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 240.032896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 247.954113] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 248.356476] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 248.369909] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 248.430423] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 264.377772] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 264.515962] Lustre: lustre-OST0001: new disk, initializing [ 264.520406] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 264.526547] Lustre: 9530:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 264.599455] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 269.937749] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 269.960441] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 270.076328] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 272.496820] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 285.656044] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 293.002641] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 300.768337] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing check_logdir /tmp/testlogs/ [ 307.045437] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing yml_node [ 312.537114] Lustre: DEBUG MARKER: Client: 2.17.57.80 [ 315.468784] Lustre: DEBUG MARKER: MDS: 2.17.57.80 [ 318.392942] Lustre: DEBUG MARKER: OSS: 2.17.57.80 [ 320.880685] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Aug 24 10:54:52 EDT 2026 [ 340.464421] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 342.153980] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 359.912637] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 359.928354] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 359.948777] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.468032] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 362.470170] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 362.484503] Lustre: Skipped 2 previous similar messages [ 362.501171] Lustre: Skipped 2 previous similar messages [ 364.901159] Lustre: server umount lustre-MDT0000 complete [ 369.056317] LustreError: 10344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583343 with bad export cookie 12818100202144484371 [ 369.059228] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 369.081524] LustreError: 10344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 372.706546] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 372.707488] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 372.736234] LustreError: 6526:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 372.767570] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 372.771719] Lustre: Skipped 1 previous similar message [ 374.757771] LustreError: 11385:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 374.780819] LustreError: 11385:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 375.631438] Lustre: server umount lustre-MDT0001 complete [ 379.746934] Lustre: server umount lustre-OST0000 complete [ 383.640435] Lustre: server umount lustre-OST0001 complete [ 391.307931] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 402.249423] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 441.471546] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 441.795656] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 441.835514] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 441.926249] Lustre: lustre-MDT0000: new disk, initializing [ 442.015762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 442.061720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 447.647993] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 459.461649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 459.621258] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 459.716841] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 459.723384] Lustre: Skipped 1 previous similar message [ 459.814754] Lustre: lustre-MDT0001: new disk, initializing [ 459.992644] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 460.031712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 460.044980] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 464.738327] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 474.703597] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 480.575204] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 487.998796] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 488.354391] Lustre: lustre-OST0000: new disk, initializing [ 488.358968] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 488.371611] Lustre: 17053:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 488.486795] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 489.956746] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 489.970722] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 490.151022] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 495.906462] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 505.985263] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 510.998312] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 511.164507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 516.204257] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 516.440529] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 520.674366] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 520.682641] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 520.715139] Lustre: Skipped 2 previous similar messages [ 520.716684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 520.734145] Lustre: Skipped 4 previous similar messages [ 525.282947] Lustre: server umount lustre-OST0000 complete [ 530.926852] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 530.932113] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 530.946530] Lustre: Skipped 1 previous similar message [ 530.970332] Lustre: Skipped 1 previous similar message [ 536.551232] Lustre: server umount lustre-MDT0000 complete [ 539.851142] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583514 with bad export cookie 12818100202144485897 [ 539.851441] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 539.856565] LustreError: 15075:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 540.255922] Lustre: server umount lustre-MDT0001 complete [ 544.578472] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:58:37 (1787583517) [ 546.927440] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 548.971219] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:58:41 (1787583521) [ 556.166932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 556.604100] LustreError: 19461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 556.693777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 560.517235] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 562.150088] LustreError: 19462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 567.266238] LustreError: 19461:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.504205] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 575.353305] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 585.647266] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 593.652426] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 601.951623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 602.095507] LustreError: 21286:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 608.195136] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 616.700449] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 628.193391] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 628.202366] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 628.221696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 628.235052] Lustre: Skipped 2 previous similar messages [ 631.452567] Lustre: server umount lustre-OST0000 complete [ 635.876321] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 635.889242] LustreError: Skipped 1 previous similar message [ 644.577610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 644.586282] Lustre: Skipped 6 previous similar messages [ 650.208147] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 650.591915] Lustre: server umount lustre-MDT0000 complete [ 653.799105] LustreError: 19462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 653.830736] LustreError: 19462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 655.152418] LustreError: 19442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787583629 with bad export cookie 12818100202144486779 [ 655.153904] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 655.181064] LustreError: 19442:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 655.704473] Lustre: server umount lustre-MDT0001 complete [ 668.523867] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 672.099575] Key type lgssc unregistered [ 672.401397] LNet: 22999:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 672.412662] LNetError: 22999:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 673.447893] LNet: Removed LNI 192.168.201.103@tcp [ 674.675813] Key type .llcrypt unregistered [ 674.687417] Key type ._llcrypt unregistered [ 699.902513] Key type ._llcrypt registered [ 699.908546] Key type .llcrypt registered [ 700.092933] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 701.433464] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 701.512333] alg: No test for adler32 (adler32-zlib) [ 702.645915] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 702.987224] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 704.712234] Key type lgssc registered [ 706.007608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 714.784303] loop: module loaded [ 714.855962] loop0: detected capacity change from 0 to 204800000 [ 715.138366] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.152969] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 715.544042] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 721.639241] loop0: detected capacity change from 0 to 204800000 [ 721.813030] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 721.834635] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 721.846948] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 727.734655] loop0: detected capacity change from 0 to 204800000 [ 733.904099] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 733.938511] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 735.216783] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 735.265531] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 735.343445] Lustre: lustre-MDT0000: new disk, initializing [ 735.440348] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 735.461983] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 741.509033] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 751.924365] loop1: detected capacity change from 0 to 204800000 [ 758.750326] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 759.151359] Lustre: lustre-OST07c6: new disk, initializing [ 759.157401] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 759.169204] Lustre: Skipped 1 previous similar message [ 759.193729] Lustre: 26207:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 759.297961] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 765.707968] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 767.616344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 767.625685] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 767.677908] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 770.786832] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 770.794115] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 770.801375] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 770.811071] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 770.824619] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 770.844351] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 789.336426] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 789.343424] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 789.351303] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 789.359599] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 789.366284] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 789.374966] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 794.440535] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 794.459209] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 794.477071] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 794.490932] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.504414] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 794.510435] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.514549] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 794.525756] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.552865] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 794.562602] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 794.572474] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 794.580438] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.879862] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 797.895459] Lustre: 25194:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 797.902884] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 797.911515] Lustre: 25194:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.935613] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 797.947275] Lustre: 25194:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 797.975653] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 797.995957] Lustre: 25194:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.023355] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 798.035760] Lustre: 25194:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 798.052873] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 798.068471] Lustre: 25194:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.441580] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 803.450121] Lustre: 25192:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 803.458666] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 803.470452] Lustre: 25192:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.480854] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 803.493672] Lustre: 25192:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.505814] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 803.518073] Lustre: 25192:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.528230] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 803.536485] Lustre: 25192:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 803.542276] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 803.549415] Lustre: 25192:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 811.490031] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 811.503928] Lustre: 25193:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 811.513909] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 811.522295] Lustre: 25193:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.533422] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 811.541631] Lustre: 25193:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.546055] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 811.553227] Lustre: 25193:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.558890] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 811.563582] Lustre: 25193:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 811.569283] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 811.574900] Lustre: 25193:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 864.739985] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 864.748581] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 869.687661] Lustre: server umount lustre-OST07c6 complete [ 880.122477] Lustre: server umount lustre-MDT0000 complete [ 901.959298] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 905.956508] Key type lgssc unregistered [ 906.252487] LNet: 28764:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 906.260957] LNetError: 28764:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 907.311687] LNet: Removed LNI 192.168.201.103@tcp [ 908.202380] Key type .llcrypt unregistered [ 908.210124] Key type ._llcrypt unregistered [ 930.578944] Key type ._llcrypt registered [ 930.584855] Key type .llcrypt registered [ 930.811316] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 932.484508] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 932.611771] alg: No test for adler32 (adler32-zlib) [ 933.741468] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 934.016245] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 935.704229] Key type lgssc registered [ 936.883206] Lustre: Echo OBD driver; http://www.lustre.org/ [ 944.444733] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 944.463246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 945.922322] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 950.462170] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 956.389717] LustreError: 30038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 956.409187] LustreError: 30038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 958.289961] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 958.776250] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 963.154955] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 970.757937] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 977.751947] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 985.018282] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 985.505590] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 990.635123] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 992.217712] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1000.203813] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1019.362172] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1019.373540] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1019.407639] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1019.413429] Lustre: Skipped 1 previous similar message [ 1024.482553] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1024.494025] Lustre: Skipped 1 previous similar message [ 1025.332955] Lustre: server umount lustre-OST0000 complete [ 1029.617198] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1029.626839] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.644198] Lustre: Skipped 2 previous similar messages [ 1029.662136] Lustre: Skipped 1 previous similar message [ 1034.728774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1034.741152] Lustre: Skipped 1 previous similar message [ 1035.618608] Lustre: server umount lustre-MDT0000 complete [ 1039.849899] LustreError: 30037:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1039.877813] LustreError: 30037:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1039.990993] LustreError: 30062:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584014 with bad export cookie 16927800217377548457 [ 1040.011951] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1040.664711] Lustre: server umount lustre-MDT0001 complete [ 1054.112547] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1057.591248] Key type lgssc unregistered [ 1058.058637] LNet: 33777:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1058.062357] LNetError: 33777:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1059.125780] LNet: Removed LNI 192.168.201.103@tcp [ 1060.273166] Key type .llcrypt unregistered [ 1060.275306] Key type ._llcrypt unregistered [ 1081.855109] Key type ._llcrypt registered [ 1081.857897] Key type .llcrypt registered [ 1081.965632] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1083.166204] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1083.301181] alg: No test for adler32 (adler32-zlib) [ 1084.417897] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 1084.653350] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1086.360320] Key type lgssc registered [ 1087.919180] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1096.715153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1096.741469] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1098.372695] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1104.682566] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1107.950237] LustreError: 35050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1107.968472] LustreError: 35050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1113.067120] LustreError: 35049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1114.652850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1115.214332] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1120.962928] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1129.899757] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1137.352407] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1145.770868] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1146.393265] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1154.397466] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1164.110901] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1167.928075] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1183.217207] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1183.233846] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1183.234264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1183.241643] Lustre: Skipped 1 previous similar message [ 1187.632935] Lustre: server umount lustre-OST0000 complete [ 1191.904898] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1191.924344] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1191.948987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1191.952667] Lustre: Skipped 1 previous similar message [ 1193.444295] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1193.444426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1193.481713] Lustre: Skipped 1 previous similar message [ 1198.564549] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1202.726229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1202.740175] Lustre: Skipped 2 previous similar messages [ 1206.240441] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1207.783231] LustreError: 35778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1207.838185] LustreError: 35778:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1208.508965] Lustre: server umount lustre-MDT0000 complete [ 1212.905452] LustreError: 35050:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.941489] LustreError: 35050:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1214.730867] LustreError: 35030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584189 with bad export cookie 2969081490473809120 [ 1214.733235] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1214.755649] LustreError: 35030:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1215.170771] Lustre: server umount lustre-MDT0001 complete [ 1228.719563] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1232.178724] Key type lgssc unregistered [ 1232.536330] LNet: 38613:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1232.550383] LNetError: 38613:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1232.572744] LNet: Removed LNI 192.168.201.103@tcp [ 1233.541792] Key type .llcrypt unregistered [ 1233.544903] Key type ._llcrypt unregistered [ 1260.547448] Key type ._llcrypt registered [ 1260.554504] Key type .llcrypt registered [ 1260.696669] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1262.324399] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1262.371522] alg: No test for adler32 (adler32-zlib) [ 1263.564313] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 1263.828275] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1265.480179] Key type lgssc registered [ 1266.607779] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1275.611292] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1275.627684] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.084652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1281.200779] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1286.638112] LustreError: 40030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1286.668088] LustreError: 40030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1289.368210] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1289.583578] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1293.707448] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1306.160776] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1313.487807] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1322.956985] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1323.720051] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.817559] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1344.263296] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1347.334320] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1357.795455] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1357.799720] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1357.827400] Lustre: Skipped 2 previous similar messages [ 1357.848995] Lustre: Skipped 2 previous similar messages [ 1360.843372] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 1362.916409] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1362.939333] Lustre: Skipped 2 previous similar messages [ 1363.529484] Lustre: server umount lustre-MDT0000 complete [ 1365.933591] LustreError: 40026:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.045632] LustreError: 40030:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1368.058083] LustreError: 40030:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1369.180256] LustreError: 40011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584343 with bad export cookie 13777294685246249436 [ 1369.183653] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1369.192945] LustreError: 40011:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1371.096580] LustreError: 42526:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1371.169213] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [ 1373.157678] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1375.706591] Lustre: server umount lustre-MDT0001 complete [ 1391.500855] Lustre: server umount lustre-OST0000 complete [ 1409.038731] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1412.244961] Key type lgssc unregistered [ 1412.703621] LNet: 43899:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1412.715591] LNetError: 43899:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1413.734072] LNet: Removed LNI 192.168.201.103@tcp [ 1414.803362] Key type .llcrypt unregistered [ 1414.805922] Key type ._llcrypt unregistered [ 1438.928138] Key type ._llcrypt registered [ 1438.929285] Key type .llcrypt registered [ 1439.016238] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1440.491205] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1440.520033] alg: No test for adler32 (adler32-zlib) [ 1441.606297] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 1441.813630] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1443.512233] Key type lgssc registered [ 1444.619748] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1452.358141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1452.372664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.875423] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1458.514931] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1464.291456] LustreError: 45318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1464.317893] LustreError: 45318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1467.602971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.938967] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1473.231878] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1481.442709] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1488.517472] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1496.215608] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1496.528070] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1499.639020] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1503.813502] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1512.410414] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1520.097961] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1520.109184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1520.132544] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1523.171211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1523.175825] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1523.198754] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1523.232443] Lustre: Skipped 2 previous similar messages [ 1525.714848] LustreError: 47140:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1525.792661] Lustre: server umount lustre-OST0000 complete [ 1544.673885] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1544.684462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.703904] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.752261] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1547.755531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.784903] Lustre: Skipped 1 previous similar message [ 1550.305870] Lustre: server umount lustre-MDT0000 complete [ 1552.869218] LustreError: 46046:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1552.885757] LustreError: 46046:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1554.783510] LustreError: 46059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584529 with bad export cookie 11624080091374405480 [ 1554.791641] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1554.801374] LustreError: 46059:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1569.251092] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1569.785946] Lustre: server umount lustre-MDT0001 complete [ 1581.975949] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1585.110426] Key type lgssc unregistered [ 1585.374240] LNet: 49031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1585.384131] LNetError: 49031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1585.404299] LNet: Removed LNI 192.168.201.103@tcp [ 1586.579423] Key type .llcrypt unregistered [ 1586.582892] Key type ._llcrypt unregistered [ 1610.208913] Key type ._llcrypt registered [ 1610.212762] Key type .llcrypt registered [ 1610.302343] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1611.191269] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1611.292992] alg: No test for adler32 (adler32-zlib) [ 1612.313939] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 1612.562981] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1614.232205] Key type lgssc registered [ 1615.231222] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1622.916245] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1622.935093] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1624.459772] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1629.280938] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1633.763083] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1633.775586] LustreError: 50446:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1637.793711] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1638.219661] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1643.109458] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1651.971131] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1658.547664] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1665.589286] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1665.903396] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1669.063944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1672.763424] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1681.822240] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1688.596585] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1692.611288] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1704.984382] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-08-2411h17m56s [ 1706.730183] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1710.052676] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1710.056855] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1710.095212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1711.148491] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1711.154684] Lustre: Skipped 3 previous similar messages [ 1714.915929] Lustre: server umount lustre-MDT0000 complete [ 1716.652361] LustreError: 50442:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1719.234609] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1722.794863] LustreError: 52940:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1722.829733] LustreError: 52940:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1726.489288] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1726.675388] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1726.959430] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1727.037696] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1727.991574] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1731.772757] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1735.826154] Lustre: DEBUG MARKER: Wait for df (36804) ... [ 1737.358763] Lustre: DEBUG MARKER: done [ 1742.307900] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1742.318257] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1742.337208] Lustre: Skipped 2 previous similar messages [ 1742.347274] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1742.354117] Lustre: Skipped 1 previous similar message [ 1743.331193] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1748.449150] LustreError: 52267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1748.463472] LustreError: 52267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1748.666604] Lustre: server umount lustre-OST0000 complete [ 1758.693831] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1758.700446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1758.700575] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1758.700580] Lustre: Skipped 2 previous similar messages [ 1758.708110] Lustre: Skipped 1 previous similar message [ 1760.460629] Lustre: server umount lustre-MDT0000 complete [ 1764.945785] LustreError: 50426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584739 with bad export cookie 14482121375354728454 [ 1764.952523] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1764.973337] LustreError: 50426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1765.414247] Lustre: server umount lustre-MDT0001 complete [ 1778.924523] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 1781.785346] Key type lgssc unregistered [ 1782.070723] LNet: 55383:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1782.079912] LNetError: 55383:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1782.090887] LNet: Removed LNI 192.168.201.103@tcp [ 1783.414933] Key type .llcrypt unregistered [ 1783.419501] Key type ._llcrypt unregistered [ 1823.393407] Key type ._llcrypt registered [ 1823.400781] Key type .llcrypt registered [ 1823.574944] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 1824.773396] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1824.857634] alg: No test for adler32 (adler32-zlib) [ 1826.126144] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 1826.457540] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 1828.120187] Key type lgssc registered [ 1829.457074] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1838.719753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1838.765124] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1840.312846] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1840.387139] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1840.654819] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1846.275373] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1854.647595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1854.743424] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1854.782380] Lustre: 57573:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1854.965970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1859.101326] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1868.977659] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1875.730850] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1883.036777] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1883.368736] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1883.699492] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1891.913338] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 1893.662128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1901.097550] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1909.469320] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-08-2411h21m21s [ 1911.160531] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1919.157146] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1921.979637] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1922.216824] LustreError: 57578:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1942.520483] Lustre: lustre-MDT0000: Client e66b4ef2-8f5a-4efb-8d56-b37fee4f0e87 (at 192.168.201.3@tcp) reconnecting [ 1962.314549] LustreError: 57578:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1964.988383] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1968.078654] Lustre: DEBUG MARKER: done [ 1975.725819] Lustre: server umount lustre-OST0000 complete [ 1980.898190] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1980.912303] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1980.916326] Lustre: Skipped 1 previous similar message [ 1981.990559] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.005387] Lustre: Skipped 1 previous similar message [ 1988.074761] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1992.161832] Lustre: 56897:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787584950/real 1787584950] req@00000000a3675404 x1874418519301632/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787584966 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1992.223926] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1992.244809] Lustre: Skipped 1 previous similar message [ 1993.317648] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1993.329506] Lustre: Skipped 2 previous similar messages [ 1994.208186] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1994.600563] Lustre: server umount lustre-MDT0000 complete [ 1997.804634] LustreError: 57564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787584972 with bad export cookie 15142412178034171382 [ 1997.818780] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1997.824446] LustreError: 57564:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1998.252895] Lustre: server umount lustre-MDT0001 complete [ 2010.197483] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2013.101696] Key type lgssc unregistered [ 2013.362486] LNet: 61905:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2013.368971] LNetError: 61905:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2014.441707] LNet: Removed LNI 192.168.201.103@tcp [ 2015.302698] Key type .llcrypt unregistered [ 2015.304831] Key type ._llcrypt unregistered [ 2063.575805] Key type ._llcrypt registered [ 2063.577765] Key type .llcrypt registered [ 2063.679984] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2065.216557] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2065.291591] alg: No test for adler32 (adler32-zlib) [ 2066.490762] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 2066.799365] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2068.480944] Key type lgssc registered [ 2069.952637] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2073.576684] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2073.610167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2075.040843] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2075.063668] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2075.253487] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.441477] Lustre: Failing over lustre-MDT0000 [ 2076.859429] Lustre: server umount lustre-MDT0000 complete [ 2085.015203] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 11:24:17 (1787585057) [ 2092.480638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2093.070957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2097.677093] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2106.494208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2106.692259] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2106.796953] Lustre: 64708:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2107.205047] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2112.318313] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2121.201213] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2128.079795] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2135.993048] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2136.244931] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2136.350899] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2144.235952] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2144.937735] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2153.455452] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2161.008640] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2170.855895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2170.906505] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2170.923128] Lustre: Skipped 1 previous similar message [ 2175.971336] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2175.980995] Lustre: Skipped 2 previous similar messages [ 2176.292110] Lustre: server umount lustre-MDT0000 complete [ 2179.693065] LustreError: 64700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585154 with bad export cookie 14005619104745419069 [ 2179.705926] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2179.712739] LustreError: 64700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2180.209857] Lustre: server umount lustre-MDT0001 complete [ 2184.506771] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2189.971594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2197.472185] Lustre: 63630:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585155/real 1787585155] req@00000000c00ae779 x1874418771995008/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787585171 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2197.509280] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2197.524903] Lustre: Skipped 2 previous similar messages [ 2199.733410] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2200.544132] Lustre: 63629:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585158/real 1787585158] req@000000000d5dc793 x1874418771995136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787585174 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2202.592416] Lustre: 63631:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585160/real 1787585160] req@00000000190fd7d5 x1874418771995264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787585176 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2204.708620] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xc25df45eedc67a20 [ 2204.726037] Lustre: MGC192.168.201.103@tcp: Connection restored to 0@lo (at 0@lo) [ 2205.325924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2205.456552] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2206.240161] Lustre: 63631:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585164/real 1787585164] req@00000000a2d043c6 x1874418771995520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787585180 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2211.305710] Lustre: 63630:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585169/real 1787585169] req@00000000d24ed656 x1874418771995648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787585185 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2211.444093] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2212.837721] LustreError: 68476:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2212.870977] LustreError: 68476:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2217.956169] LustreError: 68473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2217.985570] LustreError: 68473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2220.740885] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2221.221188] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2226.359573] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2227.898410] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2238.933089] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2246.216589] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2256.865335] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2256.886791] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2256.906756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2260.859153] Lustre: server umount lustre-MDT0000 complete [ 2262.499835] LustreError: 68473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2262.530025] LustreError: 68473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2265.376809] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585239 with bad export cookie 14005619104745421344 [ 2265.394925] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2265.678698] Lustre: server umount lustre-MDT0001 complete [ 2273.330765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2283.647190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2284.004176] Lustre: 63631:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787585242/real 1787585242] req@00000000a5453954 x1874418772024448/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1787585258 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2284.045143] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2284.056969] Lustre: Skipped 2 previous similar messages [ 2290.864154] LustreError: 66556:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2291.037432] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2291.178779] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2296.456833] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2299.361938] LustreError: 71157:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.372881] LustreError: 71157:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 2305.758976] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2311.688119] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2311.699316] Lustre: Skipped 1 previous similar message [ 2312.566230] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2322.207541] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2329.358945] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2336.737411] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2336.740484] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2336.746238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2336.756287] Lustre: Skipped 3 previous similar messages [ 2342.685347] Lustre: server umount lustre-MDT0000 complete [ 2343.395839] LustreError: 71159:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2343.420967] LustreError: 71159:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2346.399893] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585320 with bad export cookie 14005619104745422723 [ 2346.404895] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2346.406028] LustreError: 64699:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 2346.619917] Lustre: server umount lustre-MDT0001 complete [ 2352.833884] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2357.625964] Lustre: DEBUG MARKER: files compared the same [ 2370.156873] Lustre: server umount lustre-OST0000 complete [ 2387.750221] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2390.546635] Key type lgssc unregistered [ 2390.813620] LNet: 74442:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.821266] LNetError: 74442:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2390.834071] LNet: Removed LNI 192.168.201.103@tcp [ 2391.627091] Key type .llcrypt unregistered [ 2391.632745] Key type ._llcrypt unregistered [ 2416.127621] Key type ._llcrypt registered [ 2416.129492] Key type .llcrypt registered [ 2416.296894] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2417.507676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2417.750552] alg: No test for adler32 (adler32-zlib) [ 2418.942801] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 2419.147438] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2420.821331] Key type lgssc registered [ 2421.790910] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2430.773311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2430.786350] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2432.437975] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2438.305197] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 2441.701140] LustreError: 75861:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2441.716427] LustreError: 75861:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2446.834332] LustreError: 75860:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2447.141795] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2447.334898] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2452.623124] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 2460.594517] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2466.514225] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2473.733384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2474.148771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2478.269304] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2481.177885] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug +malloc all [ 2487.371239] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2504.169807] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2504.171074] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2504.185653] LustreError: Skipped 1 previous similar message [ 2504.223945] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2507.685065] Lustre: server umount lustre-OST0000 complete [ 2513.889689] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2513.921091] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2513.941459] Lustre: Skipped 1 previous similar message [ 2513.949115] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2513.961936] Lustre: Skipped 1 previous similar message [ 2519.521642] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2519.524878] Lustre: Skipped 3 previous similar messages [ 2523.616942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2523.622203] Lustre: Skipped 1 previous similar message [ 2527.200542] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2527.333510] Lustre: server umount lustre-MDT0000 complete [ 2528.740704] LustreError: 76588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2528.761538] LustreError: 76588:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2531.558671] LustreError: 75842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585505 with bad export cookie 7342368821007805346 [ 2531.566322] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2531.583126] LustreError: 75842:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2531.932828] Lustre: server umount lustre-MDT0001 complete [ 2545.617406] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2548.400478] Key type lgssc unregistered [ 2548.725854] LNet: 79416:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2548.745244] LNetError: 79416:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2548.770500] LNet: Removed LNI 192.168.201.103@tcp [ 2549.605601] Key type .llcrypt unregistered [ 2549.608368] Key type ._llcrypt unregistered [ 2581.400288] Key type ._llcrypt registered [ 2581.403502] Key type .llcrypt registered [ 2581.559202] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2583.480836] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2583.557293] alg: No test for adler32 (adler32-zlib) [ 2584.711623] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 2585.000197] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2586.752312] Key type lgssc registered [ 2587.995793] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2595.712162] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2595.747442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2611.360664] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2616.480381] LustreError: 80821:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [ 2643.234978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2650.647710] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2659.153233] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2668.035881] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2668.370399] Lustre: *** cfs_fail_loc=706, val=0*** [ 2668.908570] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.964248] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2673.490365] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2674.146948] LustreError: 80847:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2674.157131] LustreError: 80847:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2678.251354] LustreError: 81906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2678.265341] LustreError: 81906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2681.816461] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2682.423422] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2686.923990] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2694.045318] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2700.554801] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2702.822668] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2702.824606] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2702.840691] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2707.942061] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2707.954878] Lustre: Skipped 1 previous similar message [ 2709.128797] Lustre: server umount lustre-OST0000 complete [ 2718.183654] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2718.195392] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2718.200681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2718.200690] Lustre: Skipped 1 previous similar message [ 2718.230855] Lustre: Skipped 2 previous similar messages [ 2720.291850] Lustre: server umount lustre-MDT0000 complete [ 2723.303895] LustreError: 81906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2723.319990] LustreError: 81906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2723.851417] LustreError: 83683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787585698 with bad export cookie 14673027078956893001 [ 2723.853247] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2723.860073] LustreError: 83683:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2724.077046] Lustre: server umount lustre-MDT0001 complete [ 2737.946935] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 2741.655856] Key type lgssc unregistered [ 2742.238321] LNet: 84441:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2742.260267] LNetError: 84441:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2742.288527] LNet: Removed LNI 192.168.201.103@tcp [ 2743.381883] Key type .llcrypt unregistered [ 2743.383509] Key type ._llcrypt unregistered [ 2770.077059] Key type ._llcrypt registered [ 2770.078567] Key type .llcrypt registered [ 2770.166927] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2771.230674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2771.306368] alg: No test for adler32 (adler32-zlib) [ 2772.489207] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 2772.870956] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 2774.593372] Key type lgssc registered [ 2775.758467] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2783.172127] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2783.187695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2788.810951] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2798.978399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2799.533984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2804.724328] LustreError: 86554:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.754043] LustreError: 86554:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 2806.131837] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2808.806990] LustreError: 86558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2808.832944] LustreError: 86558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2813.926783] LustreError: 86557:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2813.943130] LustreError: 86557:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2815.292263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2815.687822] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2821.093061] LustreError: 86553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2821.123436] LustreError: 86553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2822.137765] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2829.283518] LustreError: 86558:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2829.301524] LustreError: 86558:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 2830.599742] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2830.623143] Lustre: MGS: Client a1256b36-0d1d-41c0-bc25-e266fddde5b0 (at 0@lo) reconnecting [ 2830.650662] LustreError: 86568:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2830.983841] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2831.076889] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2836.260611] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 2851.309329] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2851.316195] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.327191] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2851.809484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2851.825292] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2851.838963] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2856.297933] Lustre: server umount lustre-OST0000 complete [ 2862.050794] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2862.051090] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2862.063040] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2862.079871] Lustre: Skipped 1 previous similar message [ 2862.089814] Lustre: Skipped 1 previous similar message [ 2867.207077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2867.222966] Lustre: Skipped 1 previous similar message [ 2872.290503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2872.304316] Lustre: Skipped 1 previous similar message [ 2874.336528] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2874.421906] Lustre: server umount lustre-MDT0000 complete [ 2877.791815] Lustre: server umount lustre-MDT0001 complete [ 2882.549745] Lustre: server umount MGS complete [ 2898.781729] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 11:37:50 (1787585870) [ 2931.287499] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 2943.801846] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 2988.077788] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2993.435104] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3004.119414] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3004.375894] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 3004.436905] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3004.525380] Lustre: lustre-MDT0001: new disk, initializing [ 3004.628589] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3004.689240] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3004.700734] Lustre: 93731:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 3004.729732] LustreError: 93731:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 3010.133875] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3021.606947] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3021.984805] Lustre: lustre-OST0000: new disk, initializing [ 3021.990083] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3021.994362] Lustre: 94497:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3022.101572] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3029.089979] Lustre: 94784:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000022a63382 x1874419512288128/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787586019 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3029.716995] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3039.642369] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3046.753578] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3046.770820] Lustre: MGS: Client 7cccb50a-5a3e-4db1-aa16-c28bc0e11ae8 (at 0@lo) reconnecting [ 3046.836250] Lustre: 92995:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3046.861617] LustreError: 93717:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3046.872017] LustreError: 93717:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 3046.908923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3046.995083] Lustre: lustre-MDT0000: new disk, initializing [ 3047.113236] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3047.147922] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3047.179435] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3048.225605] Lustre: 95568:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000578245ae x1874419512295168/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1787586038 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3051.851637] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3053.411412] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3056.619665] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 3056.636083] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 3057.323385] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 3057.460829] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000401 [ 3082.210452] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3082.220954] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3082.247872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3082.250471] Lustre: Skipped 2 previous similar messages [ 3087.928095] Lustre: server umount lustre-OST0000 complete [ 3096.035545] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3096.047234] Lustre: Skipped 1 previous similar message [ 3096.065968] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3096.072075] LustreError: Skipped 1 previous similar message [ 3101.154284] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3101.172890] Lustre: Skipped 6 previous similar messages [ 3106.272530] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3106.363672] Lustre: server umount lustre-MDT0000 complete [ 3124.192219] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3124.445298] Lustre: server umount lustre-MDT0001 complete [ 3127.755629] Lustre: server umount MGS complete [ 3138.361158] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 11:41:50 (1787586110) [ 3154.373513] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3157.256047] Key type lgssc unregistered [ 3157.519341] LNet: 98069:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3157.527054] LNetError: 98069:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3157.543722] LNet: Removed LNI 192.168.201.103@tcp [ 3158.821420] Key type .llcrypt unregistered [ 3158.823515] Key type ._llcrypt unregistered [ 3180.058515] Key type ._llcrypt registered [ 3180.060235] Key type .llcrypt registered [ 3180.168523] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3181.688251] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3181.721286] alg: No test for adler32 (adler32-zlib) [ 3183.075746] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 3183.254239] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3185.000148] Key type lgssc registered [ 3186.555507] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3194.752968] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3194.765429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3196.080682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3199.730198] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3206.627245] LustreError: 99301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3206.660867] LustreError: 99301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3209.752495] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3209.997837] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3214.637993] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3222.575943] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3229.621453] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3236.923287] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3237.185328] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3242.306231] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:33) [ 3244.289900] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3252.569871] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3266.017957] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3266.036774] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3266.048810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3266.049988] Lustre: Skipped 1 previous similar message [ 3266.065053] Lustre: Skipped 1 previous similar message [ 3271.178696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3271.188435] Lustre: Skipped 1 previous similar message [ 3271.535359] Lustre: server umount lustre-OST0000 complete [ 3276.258073] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3276.264322] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3276.282784] Lustre: Skipped 1 previous similar message [ 3276.322882] Lustre: Skipped 1 previous similar message [ 3281.380238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3281.743438] Lustre: server umount lustre-MDT0000 complete [ 3285.737470] LustreError: 100035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586260 with bad export cookie 13406520428188789278 [ 3285.750787] LustreError: 100035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3285.753600] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3286.246475] Lustre: server umount lustre-MDT0001 complete [ 3297.087439] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3300.136957] Key type lgssc unregistered [ 3300.545731] LNet: 102854:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3300.551248] LNetError: 102854:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3300.566817] LNet: Removed LNI 192.168.201.103@tcp [ 3301.439727] Key type .llcrypt unregistered [ 3301.441806] Key type ._llcrypt unregistered [ 3319.167092] Key type ._llcrypt registered [ 3319.171589] Key type .llcrypt registered [ 3319.334181] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3320.667700] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3320.721147] alg: No test for adler32 (adler32-zlib) [ 3321.955333] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 3322.394397] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3324.104250] Key type lgssc registered [ 3325.553958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3334.263264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3334.268256] LustreError: 103937:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3334.272589] Lustre: Skipped 1 previous similar message [ 3334.272611] LustreError: 103938:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3334.300719] LustreError: 103937:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=5000 [ 3334.307662] LustreError: 103937:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3334.313304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3334.317260] LustreError: 103937:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3335.897135] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3345.459464] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3345.759474] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3351.024512] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3358.405155] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing lsmod [ 3361.463664] LustreError: 105140:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 3361.500517] LustreError: 105147:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 3361.505514] LustreError: 105140:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4975 [ 3361.511254] LustreError: 105140:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3361.526685] LustreError: 105140:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3361.531983] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3361.972366] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3362.994980] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:65) [ 3366.369844] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3366.374131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3366.390102] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.418880] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3368.421933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3368.427340] Lustre: Skipped 1 previous similar message [ 3368.452349] Lustre: Skipped 2 previous similar messages [ 3371.492449] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3371.515328] Lustre: Skipped 2 previous similar messages [ 3376.629938] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3379.169891] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3379.480902] Lustre: server umount lustre-MDT0000 complete [ 3381.738371] LustreError: 103973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3381.777391] LustreError: 103973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 3383.683026] LustreError: 103951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586358 with bad export cookie 13193159918885726547 [ 3383.686925] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3383.700897] LustreError: 103951:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3383.840258] LustreError: 104407:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000f9ce3e0f x1874420087938304/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3384.203378] Lustre: server umount lustre-MDT0001 complete [ 3399.334346] Lustre: server umount lustre-OST0000 complete [ 3407.208913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3407.870474] LustreError: 106210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3407.990618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3412.411585] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3413.486207] LustreError: 106211:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3418.593121] LustreError: 106210:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3422.452276] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3422.798958] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3428.666450] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3438.200113] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3444.153869] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3451.868990] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3452.173928] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3453.245129] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:3 to 0x240000401:97) [ 3459.006335] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3465.969687] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3484.146738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3484.156444] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3484.201610] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3484.216686] Lustre: Skipped 3 previous similar messages [ 3485.879976] Lustre: server umount lustre-OST0000 complete [ 3494.370209] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3494.370298] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3494.371779] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3494.373946] LustreError: Skipped 1 previous similar message [ 3494.386356] Lustre: Skipped 2 previous similar messages [ 3496.822717] Lustre: server umount lustre-MDT0000 complete [ 3499.489255] LustreError: 106205:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3499.510241] LustreError: 106205:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3501.213111] LustreError: 106191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586475 with bad export cookie 13193159918885727478 [ 3501.219503] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3501.222602] LustreError: 106191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3501.668239] Lustre: server umount lustre-MDT0001 complete [ 3514.883434] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3518.318466] Key type lgssc unregistered [ 3518.651472] LNet: 109771:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3518.686588] LNetError: 109771:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3518.714262] LNet: Removed LNI 192.168.201.103@tcp [ 3519.511850] Key type .llcrypt unregistered [ 3519.514849] Key type ._llcrypt unregistered [ 3546.430821] Key type ._llcrypt registered [ 3546.434472] Key type .llcrypt registered [ 3546.575475] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3548.155259] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3548.186300] alg: No test for adler32 (adler32-zlib) [ 3549.398453] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 3549.567493] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3551.240248] Key type lgssc registered [ 3552.260166] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3559.610942] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3559.649413] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3561.098848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3565.424895] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3570.674663] LustreError: 111192:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3570.694750] LustreError: 111192:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3575.788281] LustreError: 111193:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3576.197430] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3576.614935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3581.877823] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3589.897621] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3595.748323] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3602.647492] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3603.201645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3606.280411] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:99 to 0x240000401:129) [ 3610.722354] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3619.337299] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3625.394346] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3629.434148] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3637.218073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3637.224979] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3637.251082] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3637.257811] Lustre: Skipped 1 previous similar message [ 3637.276350] Lustre: Skipped 1 previous similar message [ 3640.041790] Lustre: server umount lustre-OST0000 complete [ 3647.458356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3647.481328] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3647.487583] Lustre: Skipped 1 previous similar message [ 3652.588482] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3652.605692] Lustre: Skipped 1 previous similar message [ 3656.673699] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3656.675293] Lustre: Skipped 1 previous similar message [ 3658.208195] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3658.349485] Lustre: server umount lustre-MDT0000 complete [ 3661.812308] LustreError: 113034:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3661.840146] LustreError: 113034:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3663.256967] LustreError: 111927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586637 with bad export cookie 12098611122165237841 [ 3663.268072] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3663.275336] LustreError: 111927:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3664.026865] Lustre: server umount lustre-MDT0001 complete [ 3680.924794] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3685.414432] Key type lgssc unregistered [ 3685.757897] LNet: 114837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3685.768946] LNetError: 114837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3685.787181] LNet: Removed LNI 192.168.201.103@tcp [ 3687.128450] Key type .llcrypt unregistered [ 3687.131483] Key type ._llcrypt unregistered [ 3706.504732] Key type ._llcrypt registered [ 3706.506699] Key type .llcrypt registered [ 3706.613474] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3707.944395] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3708.235800] alg: No test for adler32 (adler32-zlib) [ 3709.393729] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 3709.636341] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3711.328560] Key type lgssc registered [ 3712.277442] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3720.539294] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3720.561909] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3722.109346] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3727.701672] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3732.451577] LustreError: 116063:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3732.464448] LustreError: 116063:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3737.336545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3737.574681] LustreError: 116064:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3737.819147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3743.344559] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3753.206885] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3759.652470] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3767.127785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3767.487311] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3773.535882] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3782.765403] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3792.056566] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:131 to 0x240000401:161) [ 3804.648414] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3804.658331] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3804.679703] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3807.713212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3807.720479] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3807.747540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3810.142923] Lustre: server umount lustre-OST0000 complete [ 3814.887143] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3814.896406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3814.908985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3814.911740] Lustre: Skipped 1 previous similar message [ 3817.956580] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3817.961177] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.073970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3823.085762] Lustre: Skipped 2 previous similar messages [ 3829.221768] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3829.422762] Lustre: server umount lustre-MDT0000 complete [ 3833.316468] LustreError: 116058:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3833.334226] LustreError: 116058:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3833.921923] LustreError: 116045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787586808 with bad export cookie 3296733361973773802 [ 3833.923967] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3833.929487] LustreError: 116045:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3834.309665] Lustre: server umount lustre-MDT0001 complete [ 3846.980995] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 3850.072588] Key type lgssc unregistered [ 3850.369416] LNet: 119622:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3850.382443] LNetError: 119622:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3851.449147] LNet: Removed LNI 192.168.201.103@tcp [ 3852.444434] Key type .llcrypt unregistered [ 3852.446129] Key type ._llcrypt unregistered [ 3878.135108] Key type ._llcrypt registered [ 3878.138393] Key type .llcrypt registered [ 3878.266476] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 3879.606236] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3879.802714] alg: No test for adler32 (adler32-zlib) [ 3880.970968] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 3881.352446] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 3883.120316] Key type lgssc registered [ 3884.654735] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3893.476390] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3893.491210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3895.074055] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3900.143437] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3905.513404] LustreError: 121039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3905.527041] LustreError: 121039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3908.555782] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3909.049822] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3914.920868] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3924.068940] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3930.687500] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3938.896018] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3939.449026] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3940.478833] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:163 to 0x240000401:193) [ 3947.269585] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 3955.546537] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3963.812937] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3967.857097] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3967.867362] Lustre: Skipped 2 previous similar messages [ 3969.676828] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3969.711862] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3972.409736] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3972.415375] Lustre: Skipped 2 previous similar messages [ 3976.103096] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3976.116573] Lustre: Skipped 1 previous similar message [ 3984.031992] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.3@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3984.054882] Lustre: Skipped 2 previous similar messages [ 3992.191643] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.3@tcp 192.168.0.[2,10]@tcp [ 3996.957321] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.3/24@tcp in log lustre-MDT0000 [ 3996.963265] Lustre: Skipped 2 previous similar messages [ 3999.488385] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.3/24@tcp [ 3999.501218] Lustre: Skipped 1 previous similar message [ 4009.958095] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4009.958730] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4009.964116] LustreError: Skipped 2 previous similar messages [ 4010.010488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4010.465316] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4010.476403] LustreError: Skipped 3 previous similar messages [ 4010.481187] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4015.586714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4015.602073] Lustre: Skipped 2 previous similar messages [ 4020.708344] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4020.721933] Lustre: Skipped 1 previous similar message [ 4024.288700] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4024.538560] Lustre: server umount lustre-OST0000 complete [ 4030.946603] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4030.958076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4035.026881] Lustre: server umount lustre-MDT0000 complete [ 4036.071041] LustreError: 121038:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4036.095264] LustreError: 121038:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4039.019777] LustreError: 121773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587013 with bad export cookie 15444306854092756075 [ 4039.035471] LustreError: 121773:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4039.038167] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4039.375137] Lustre: server umount lustre-MDT0001 complete [ 4051.367682] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4054.793858] Key type lgssc unregistered [ 4055.057379] LNet: 125945:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4055.062634] LNetError: 125945:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4055.088339] LNet: Removed LNI 192.168.201.103@tcp [ 4056.049480] Key type .llcrypt unregistered [ 4056.051165] Key type ._llcrypt unregistered [ 4088.464573] Key type ._llcrypt registered [ 4088.473727] Key type .llcrypt registered [ 4088.699360] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4089.936620] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4090.232681] alg: No test for adler32 (adler32-zlib) [ 4091.214544] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 4091.408541] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4093.072189] Key type lgssc registered [ 4094.381696] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4102.595914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4102.654503] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4104.663637] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4104.681121] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4104.716440] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.3/24@tcp [ 4110.390072] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4114.914255] LustreError: 127512:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4114.939649] LustreError: 127512:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4120.035429] LustreError: 127511:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4120.669036] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4121.149587] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4121.188559] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.3/24@tcp [ 4121.251780] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4126.872976] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4136.619305] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4144.357604] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4152.338656] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4152.790789] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.093569] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4168.650373] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4175.357126] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x240000401:201 to 0x240000401:225) [ 4185.571785] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4185.582637] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4185.583940] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4190.749478] Lustre: server umount lustre-OST0000 complete [ 4195.810388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4195.814946] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4195.836355] Lustre: Skipped 2 previous similar messages [ 4195.858096] Lustre: Skipped 2 previous similar messages [ 4200.929993] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4201.567470] Lustre: server umount lustre-MDT0000 complete [ 4205.967254] LustreError: 127492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587180 with bad export cookie 18431464966861863009 [ 4205.968532] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4205.991887] LustreError: 127492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4206.856520] Lustre: server umount lustre-MDT0001 complete [ 4218.997888] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4222.310494] Key type lgssc unregistered [ 4222.594752] LNet: 131103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4222.605362] LNetError: 131103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4223.654061] LNet: Removed LNI 192.168.201.103@tcp [ 4225.007097] Key type .llcrypt unregistered [ 4225.010991] Key type ._llcrypt unregistered [ 4254.209939] Key type ._llcrypt registered [ 4254.217793] Key type .llcrypt registered [ 4254.390876] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4271.603324] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4274.047392] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4274.070188] alg: No test for adler32 (adler32-zlib) [ 4275.178200] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 4275.454674] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4277.120158] Key type lgssc registered [ 4278.300957] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4313.909281] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4313.959343] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4315.230653] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4315.255790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4315.332454] Lustre: lustre-MDT0000: new disk, initializing [ 4315.433848] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4315.465147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4320.305775] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4330.695689] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4330.896589] Lustre: 134517:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4330.961151] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4330.965150] Lustre: Skipped 1 previous similar message [ 4331.038705] Lustre: lustre-MDT0001: new disk, initializing [ 4331.146271] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4331.217435] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4331.228387] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4335.757097] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4346.366458] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4351.682236] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4360.405384] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4360.849451] Lustre: lustre-OST0000: new disk, initializing [ 4360.852143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4360.885804] Lustre: 136485:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4361.121414] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4369.014581] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4370.083658] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4370.097900] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4370.241568] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4379.909675] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4385.429824] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4385.784517] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4392.045395] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4392.277328] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4396.002527] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4396.008257] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4396.062434] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4396.084054] Lustre: Skipped 1 previous similar message [ 4401.053368] Lustre: server umount lustre-OST0000 complete [ 4408.301090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4408.315581] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4408.338941] Lustre: Skipped 1 previous similar message [ 4408.351762] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4411.364569] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4411.379487] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4413.088932] Lustre: server umount lustre-MDT0000 complete [ 4417.592728] LustreError: 134512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587391 with bad export cookie 17429484405124097790 [ 4417.602466] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4417.612740] LustreError: 134512:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4418.031679] Lustre: server umount lustre-MDT0001 complete [ 4427.172702] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4427.474175] LustreError: 138703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4427.492896] LustreError: 138703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 4427.559427] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4432.871652] LustreError: 138704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4433.599109] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4436.972444] LustreError: 138703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4442.083253] LustreError: 138704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.736937] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4444.186762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4449.834784] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4458.973706] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4466.306373] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4473.900896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4474.381891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4483.335978] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4492.615026] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4497.812708] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4498.181713] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4503.911505] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4504.169624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4513.404380] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4513.609881] Lustre: lustre-OST0001: new disk, initializing [ 4513.613337] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4513.624363] Lustre: 142075:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4513.798402] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4520.824705] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4523.588158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4523.595986] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4523.698863] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4531.131632] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4531.347828] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4536.540985] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4536.783971] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4542.995896] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0970a10b800.ost_server_uuid 50 [ 4544.815387] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa0970a10b800.ost_server_uuid in IDLE state after 0 sec [ 4554.746923] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4554.750604] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4554.757827] LustreError: Skipped 1 previous similar message [ 4554.758273] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4554.758277] Lustre: Skipped 1 previous similar message [ 4554.817833] Lustre: Skipped 1 previous similar message [ 4559.847047] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4559.860702] Lustre: Skipped 1 previous similar message [ 4560.533949] Lustre: server umount lustre-OST0001 complete [ 4564.970342] LustreError: 140529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4565.013331] LustreError: 140529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4568.546094] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4568.559324] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4568.572582] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4572.668748] Lustre: server umount lustre-OST0000 complete [ 4580.325238] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4580.341427] Lustre: Skipped 1 previous similar message [ 4584.311204] Lustre: server umount lustre-MDT0000 complete [ 4585.450331] LustreError: 138703:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4585.481481] LustreError: 138703:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4588.421403] LustreError: 138685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587562 with bad export cookie 17429484405124098672 [ 4588.430189] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4588.432082] LustreError: 138685:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4588.920804] Lustre: server umount lustre-MDT0001 complete [ 4611.051202] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4614.855610] Key type lgssc unregistered [ 4615.442942] LNet: 145082:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4615.452347] LNetError: 145082:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4616.509113] LNet: Removed LNI 192.168.201.103@tcp [ 4618.173753] Key type .llcrypt unregistered [ 4618.176887] Key type ._llcrypt unregistered [ 4665.198758] Key type ._llcrypt registered [ 4665.200756] Key type .llcrypt registered [ 4665.290885] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 4680.623396] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4682.123375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4682.157168] alg: No test for adler32 (adler32-zlib) [ 4683.207592] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 4683.441333] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4685.120195] Key type lgssc registered [ 4686.328795] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4718.858829] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4718.909859] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4720.128251] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4720.151837] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4720.229154] Lustre: lustre-MDT0000: new disk, initializing [ 4720.333403] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4720.345873] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4725.891755] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4736.692595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4736.791237] Lustre: 149236:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4736.827810] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4736.831222] Lustre: Skipped 1 previous similar message [ 4736.910520] Lustre: lustre-MDT0001: new disk, initializing [ 4736.995493] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4737.044461] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4737.051870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4742.992418] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4753.151435] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4759.277276] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4766.835418] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4767.208432] Lustre: lustre-OST0000: new disk, initializing [ 4767.214335] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4767.224223] Lustre: 151208:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4767.339019] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4773.621790] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4774.002738] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4774.017992] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4774.150154] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4783.662732] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4789.661867] Lustre: Failing over lustre-OST0000 [ 4789.739462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4789.743355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4789.958492] Lustre: server umount lustre-OST0000 complete [ 4808.442355] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4808.863493] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4808.873841] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4809.896903] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4815.513542] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4817.355812] Lustre: lustre-OST0000: Recovery over after 0:08, of 3 clients 3 recovered and 0 were evicted. [ 4817.360866] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 4819.575313] Lustre: Failing over lustre-MDT0000 [ 4819.874153] Lustre: server umount lustre-MDT0000 complete [ 4820.449821] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4820.453858] LustreError: 149249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4820.475479] Lustre: Skipped 2 previous similar messages [ 4820.493798] LustreError: 149249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4822.460298] LustreError: 150301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4822.483221] LustreError: 150301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4825.572857] LustreError: 149246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4825.596868] LustreError: 149246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4830.698906] LustreError: 150301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4830.720123] LustreError: 150301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4835.809995] Lustre: 147462:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787587794/real 1787587794] req@00000000d901624e x1874421515073280/t0(0) o400->MGC192.168.201.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787587810 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4835.820295] LustreError: 149249:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4835.843437] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4835.879432] LustreError: 149249:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 4838.589797] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4846.054507] LustreError: 150301:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4846.058101] LustreError: 147461:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ec256409 x1874421515078784/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4846.072025] LustreError: 150301:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 8 previous similar messages [ 4846.284370] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4846.333274] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4848.048540] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4851.206619] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4851.688497] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4851.696440] Lustre: Skipped 1 previous similar message [ 4851.708428] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4851.777904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4861.931774] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4861.960160] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4861.971480] Lustre: Skipped 1 previous similar message [ 4866.533969] Lustre: server umount lustre-OST0000 complete [ 4872.163640] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4872.179325] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4872.193387] Lustre: Skipped 2 previous similar messages [ 4872.205401] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4872.206591] Lustre: Skipped 1 previous similar message [ 4872.243148] LustreError: Skipped 1 previous similar message [ 4877.080915] Lustre: server umount lustre-MDT0000 complete [ 4877.281818] LustreError: 149250:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4877.307338] LustreError: 149250:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4880.792955] LustreError: 153008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787587855 with bad export cookie 13600044263489856261 [ 4880.797724] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4880.806644] LustreError: 153008:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4881.148059] Lustre: server umount lustre-MDT0001 complete [ 4892.466362] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 4896.181701] Key type lgssc unregistered [ 4896.705748] LNet: 154868:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4896.716169] LNetError: 154868:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4896.748371] LNet: Removed LNI 192.168.201.103@tcp [ 4897.726071] Key type .llcrypt unregistered [ 4897.727639] Key type ._llcrypt unregistered [ 4936.121713] Key type ._llcrypt registered [ 4936.132512] Key type .llcrypt registered [ 4936.318133] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 4937.628171] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4937.662729] alg: No test for adler32 (adler32-zlib) [ 4938.934291] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 4939.348570] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 4941.097394] Key type lgssc registered [ 4942.158894] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4950.868814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4950.884255] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4952.327075] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4957.884562] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 4961.761621] LustreError: 156393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4961.781953] LustreError: 156393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4966.881173] LustreError: 156392:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4967.769543] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4972.003697] LustreError: 156393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4972.143389] LustreError: 156421:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4972.153142] LustreError: 156421:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4972.162599] LustreError: 156421:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 20, retries 0, failed: rc = -5 [ 4972.619323] Lustre: server umount lustre-MDT0000 complete [ 4989.422629] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5000.214966] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5039.269466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5039.630718] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5039.667269] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5039.738284] Lustre: lustre-MDT0000: new disk, initializing [ 5039.856288] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5039.916387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5045.573625] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5057.895643] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5058.011521] Lustre: 159725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5058.060972] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5058.065429] Lustre: Skipped 1 previous similar message [ 5058.152619] Lustre: lustre-MDT0001: new disk, initializing [ 5058.260803] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5058.304661] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5058.320834] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5063.516332] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5073.730885] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5079.601933] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5086.360449] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5086.721146] Lustre: lustre-OST0000: new disk, initializing [ 5086.732216] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5086.742437] Lustre: 161694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5086.854427] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5093.823852] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5095.498441] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5095.510750] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5095.774519] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5104.004626] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5108.914322] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5110.576664] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 12:14:43 (1787588083) [ 5346.786204] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5346.802657] Lustre: Skipped 1 previous similar message [ 5346.808244] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5346.810709] Lustre: Skipped 1 previous similar message [ 5351.908055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5351.918248] Lustre: Skipped 2 previous similar messages [ 5352.732097] Lustre: server umount lustre-MDT0000 complete [ 5356.294705] LustreError: 159716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588330 with bad export cookie 3848532581622644071 [ 5356.296916] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5356.309979] LustreError: 159716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5357.025551] LustreError: 161106:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5357.026728] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5357.053511] Lustre: Skipped 1 previous similar message [ 5357.057177] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5361.058704] LustreError: 159731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5361.060731] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5362.687745] Lustre: server umount lustre-MDT0001 complete [ 5373.156190] Lustre: server umount lustre-OST0000 complete [ 5382.747847] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5393.959300] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5433.155210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5433.608798] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5433.641885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5433.726786] Lustre: lustre-MDT0000: new disk, initializing [ 5433.797790] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5433.816235] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5438.629351] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5448.360401] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5448.469744] Lustre: 166357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5448.511610] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5448.514308] Lustre: Skipped 1 previous similar message [ 5448.625372] Lustre: lustre-MDT0001: new disk, initializing [ 5448.717861] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5448.763919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5448.773041] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5453.407822] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5463.359125] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5469.313049] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5476.318950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5476.723376] Lustre: lustre-OST0000: new disk, initializing [ 5476.726772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5476.731253] Lustre: 168324:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5476.810310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5482.591315] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5482.616168] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5482.783411] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5483.797195] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5492.195665] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5495.993477] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5496.173802] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5501.398635] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5501.657444] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5508.585777] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5508.587495] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5508.598269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5508.600291] LustreError: Skipped 1 previous similar message [ 5508.614339] Lustre: Skipped 1 previous similar message [ 5518.304509] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5518.501882] Lustre: server umount lustre-OST0000 complete [ 5523.941975] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5523.950134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5523.957128] Lustre: Skipped 1 previous similar message [ 5523.967306] Lustre: Skipped 4 previous similar messages [ 5528.714548] Lustre: server umount lustre-MDT0000 complete [ 5529.057206] LustreError: 167864:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5529.067383] LustreError: 167864:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5532.721635] LustreError: 166350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588507 with bad export cookie 3848532581622802901 [ 5532.726070] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5532.732586] LustreError: 166350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5533.102721] Lustre: server umount lustre-MDT0001 complete [ 5542.677568] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:21:55 (1787588515) [ 5559.166235] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5568.715254] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5606.587732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5606.953315] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5606.994593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5607.112210] Lustre: lustre-MDT0000: new disk, initializing [ 5607.221744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5607.245406] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5612.905136] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5624.845543] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5625.079568] Lustre: 173082:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5625.379337] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5630.006726] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5639.589340] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5644.854518] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5651.903572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5652.315657] Lustre: lustre-OST0000: new disk, initializing [ 5652.318595] Lustre: Skipped 1 previous similar message [ 5652.322332] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5652.327859] Lustre: Skipped 2 previous similar messages [ 5652.332408] Lustre: 175057:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5659.139782] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5659.225612] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5659.235408] Lustre: Skipped 1 previous similar message [ 5659.246564] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5659.360101] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5668.473626] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5674.982124] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5674.998869] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5675.001409] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5689.312441] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5689.606995] Lustre: server umount lustre-OST0000 complete [ 5695.458285] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5695.476034] Lustre: Skipped 2 previous similar messages [ 5699.872601] Lustre: server umount lustre-MDT0000 complete [ 5700.577964] LustreError: 173090:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5700.605703] LustreError: 173090:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5703.308326] LustreError: 173073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588677 with bad export cookie 3848532581622803783 [ 5703.316139] LustreError: 173073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5703.317450] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5703.646388] Lustre: server umount lustre-MDT0001 complete [ 5711.212255] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 12:24:43 (1787588683) [ 5728.298302] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 5739.616253] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5775.857809] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5776.071235] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5776.088688] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5776.185274] Lustre: lustre-MDT0000: new disk, initializing [ 5776.268838] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5776.280378] Lustre: Skipped 2 previous similar messages [ 5776.318581] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5781.248460] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5792.641370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5792.796702] Lustre: 179383:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5793.152201] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5797.392530] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5807.363332] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5814.183916] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5822.007690] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5822.280463] Lustre: 181359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5828.886703] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5831.252264] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5831.363075] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5838.160887] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5847.010906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5847.014343] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5847.017649] LustreError: Skipped 2 previous similar messages [ 5847.018490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5847.030996] Lustre: Skipped 1 previous similar message [ 5847.043148] Lustre: Skipped 8 previous similar messages [ 5852.021941] Lustre: server umount lustre-OST0000 complete [ 5861.742947] Lustre: server umount lustre-MDT0000 complete [ 5862.376676] LustreError: 180755:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5862.409403] LustreError: 180755:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5865.321348] LustreError: 179373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588839 with bad export cookie 3848532581622804868 [ 5865.325255] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5865.332080] LustreError: 179373:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5865.754221] Lustre: server umount lustre-MDT0001 complete [ 5875.290685] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 5877.846131] Key type lgssc unregistered [ 5878.131901] LNet: 183219:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5878.145821] LNetError: 183219:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5878.163049] LNet: Removed LNI 192.168.201.103@tcp [ 5878.924707] Key type .llcrypt unregistered [ 5878.927143] Key type ._llcrypt unregistered [ 5899.831557] Key type ._llcrypt registered [ 5899.832849] Key type .llcrypt registered [ 5899.925534] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 5900.764179] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5900.847071] alg: No test for adler32 (adler32-zlib) [ 5901.832270] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 5902.031564] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 5903.688202] Key type lgssc registered [ 5904.828837] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5912.321706] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5912.336595] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5913.764532] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5918.381846] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5924.327237] LustreError: 184635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5924.364856] LustreError: 184635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5926.583671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5927.040352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5931.430791] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5938.049870] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5942.746977] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5948.977821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5949.282036] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5955.409436] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 5959.571065] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5962.372506] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5983.209188] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5983.212993] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5983.225196] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5985.253729] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5985.255547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5985.294370] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5986.887299] Lustre: server umount lustre-OST0000 complete [ 5993.441610] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5993.451382] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5993.477800] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5993.484996] Lustre: Skipped 1 previous similar message [ 5995.494963] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5995.496470] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.713774] Lustre: server umount lustre-MDT0000 complete [ 6000.376926] LustreError: 184615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787588974 with bad export cookie 14260099412978132400 [ 6000.378600] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6000.386842] LustreError: 184615:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6000.655661] Lustre: server umount lustre-MDT0001 complete [ 6011.704314] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 6014.607215] Key type lgssc unregistered [ 6014.879861] LNet: 188191:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6014.899833] LNetError: 188191:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6014.920535] LNet: Removed LNI 192.168.201.103@tcp [ 6015.785427] Key type .llcrypt unregistered [ 6015.787279] Key type ._llcrypt unregistered [ 6040.433996] Key type ._llcrypt registered [ 6040.435909] Key type .llcrypt registered [ 6040.531059] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6041.974718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6042.033319] alg: No test for adler32 (adler32-zlib) [ 6043.133607] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 6043.527281] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 6045.200571] Key type lgssc registered [ 6046.665052] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6055.430448] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6055.450904] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6056.892588] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6062.043505] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6067.172384] LustreError: 189608:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6067.187517] LustreError: 189608:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6071.135886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6071.699343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6076.906657] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6084.141147] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6090.633256] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6099.591383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6100.063204] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6102.123822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6108.459336] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6116.326157] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6122.976866] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6122.981059] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6123.004015] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6123.966918] Lustre: lustre-OST0000: Not available for connect from 192.168.201.3@tcp (stopping) [ 6127.073388] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6127.078673] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6127.093139] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.126033] Lustre: Skipped 1 previous similar message [ 6132.194080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6132.213205] Lustre: Skipped 2 previous similar messages [ 6137.313171] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6137.323676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6137.326391] Lustre: Skipped 1 previous similar message [ 6137.598407] Lustre: server umount lustre-OST0000 complete [ 6146.127983] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa09709ae4800.ost_server_uuid 50 [ 6148.103294] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6162.933404] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6162.936296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6162.947645] Lustre: Skipped 1 previous similar message [ 6162.965209] Lustre: Skipped 1 previous similar message [ 6169.157065] Lustre: server umount lustre-MDT0000 complete [ 6172.131787] LustreError: 189609:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6172.149945] LustreError: 189609:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6172.882417] LustreError: 189588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589147 with bad export cookie 251429156524309421 [ 6172.884970] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6172.890678] LustreError: 189588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6173.253880] Lustre: server umount lustre-MDT0001 complete [ 6182.358830] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 12:32:34 (1787589154) [ 6189.500369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6190.054131] LustreError: 193529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6190.169181] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6195.175665] LustreError: 193530:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6195.370414] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6200.305362] LustreError: 193529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6204.585521] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6204.941390] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6209.919687] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6217.977430] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6224.022520] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6231.454792] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6231.804520] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6234.914855] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6239.177217] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6247.678656] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6255.767732] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6256.010556] Lustre: lustre-OST0001: new disk, initializing [ 6256.013799] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6256.018877] Lustre: 196369:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6256.180348] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6261.878392] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6261.886323] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6261.988507] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6263.790735] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6273.968116] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6277.611802] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6277.631317] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6277.633360] Lustre: Skipped 2 previous similar messages [ 6283.409913] Lustre: server umount lustre-OST0000 complete [ 6285.244946] LustreError: 195351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6290.634549] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6290.888387] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6295.339414] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6295.492915] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6303.209530] LustreError: 195351:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6303.236882] LustreError: 195351:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 6313.445687] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6313.468025] Lustre: Skipped 1 previous similar message [ 6313.471394] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6313.476392] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6313.484473] Lustre: Skipped 4 previous similar messages [ 6315.650502] Lustre: server umount lustre-OST0001 complete [ 6322.657410] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6322.660100] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6322.663996] Lustre: Skipped 1 previous similar message [ 6326.048129] Lustre: server umount lustre-MDT0000 complete [ 6330.220110] LustreError: 193511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589304 with bad export cookie 251429156524310492 [ 6330.228860] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6330.233862] LustreError: 193511:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6330.586445] Lustre: server umount lustre-MDT0001 complete [ 6354.645843] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 12:35:27 (1787589327) [ 6360.844695] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6361.141725] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6361.182249] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6361.409788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6366.542618] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6375.538905] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6375.642990] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6375.701937] Lustre: 199809:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6381.043417] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6390.788566] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6397.413223] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6404.940823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6405.227882] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6412.687448] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6412.974108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6420.732681] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6428.412261] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6428.483249] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6428.770278] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6428.778358] Lustre: Skipped 2 previous similar messages [ 6436.131561] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6436.506473] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6444.190294] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6448.610738] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6448.627076] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6448.654246] Lustre: Skipped 1 previous similar message [ 6448.661653] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6448.671277] Lustre: Skipped 3 previous similar messages [ 6453.651322] Lustre: server umount lustre-OST0000 complete [ 6455.224689] LustreError: 201659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6455.241469] LustreError: 201659:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6472.162691] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6472.171165] LustreError: Skipped 1 previous similar message [ 6476.473468] Lustre: server umount lustre-OST0001 complete [ 6482.920516] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6482.936139] Lustre: Skipped 4 previous similar messages [ 6495.200199] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6495.410624] Lustre: server umount lustre-MDT0000 complete [ 6499.295295] LustreError: 203576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589473 with bad export cookie 251429156524311899 [ 6499.297757] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6499.304030] LustreError: 203576:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6499.680502] Lustre: server umount lustre-MDT0001 complete [ 6525.720937] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 12:38:18 (1787589498) [ 6542.971357] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 6553.042824] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 6590.554389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6590.812782] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6590.851168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6590.969927] Lustre: lustre-MDT0000: new disk, initializing [ 6591.103744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6591.122308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6596.535380] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6607.406712] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6607.654982] Lustre: 207905:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6607.702789] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6607.706903] Lustre: Skipped 1 previous similar message [ 6607.777397] Lustre: lustre-MDT0001: new disk, initializing [ 6607.924653] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6607.942914] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6613.402956] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6623.634782] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6630.030744] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6636.909157] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6637.154425] Lustre: lustre-OST0000: new disk, initializing [ 6637.158794] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6637.165403] Lustre: 209878:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6643.831767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6643.859747] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6643.927057] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6644.051368] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6653.403667] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6658.280287] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6658.483170] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6663.341995] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6663.585488] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6669.800569] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6669.809745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6669.819075] LustreError: Skipped 1 previous similar message [ 6669.837394] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6669.844866] Lustre: Skipped 1 previous similar message [ 6669.866854] Lustre: Skipped 13 previous similar messages [ 6671.826366] Lustre: server umount lustre-OST0000 complete [ 6684.442938] Lustre: server umount lustre-MDT0000 complete [ 6685.155425] LustreError: 209326:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6685.178532] LustreError: 209326:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 6688.149939] LustreError: 209877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589662 with bad export cookie 251429156524313684 [ 6688.151774] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6688.160980] LustreError: 209877:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6688.499097] Lustre: server umount lustre-MDT0001 complete [ 6697.163427] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6701.543854] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6709.461512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6714.888045] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6721.847819] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6726.427775] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6732.724343] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6741.154234] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6751.786682] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6757.072781] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6757.247301] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6763.240178] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6763.571760] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6766.052584] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6766.065246] LustreError: Skipped 1 previous similar message [ 6780.384782] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6780.640318] Lustre: server umount lustre-OST0000 complete [ 6788.011515] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6793.541738] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6822.637488] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6829.430266] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6834.284436] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6834.305158] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6834.306536] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 6834.330376] Lustre: Skipped 1 previous similar message [ 6837.457815] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6841.713996] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 2 sec [ 6850.019770] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6850.029609] Lustre: Skipped 4 previous similar messages [ 6850.039147] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6850.044682] LustreError: Skipped 2 previous similar messages [ 6852.707152] Lustre: server umount lustre-OST0000 complete [ 6862.936809] Lustre: server umount lustre-MDT0000 complete [ 6865.383723] LustreError: 212093:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6865.417979] LustreError: 212093:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 6867.648721] LustreError: 212074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787589842 with bad export cookie 251429156524314559 [ 6867.649427] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6867.675320] LustreError: 212074:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6868.682837] Lustre: server umount lustre-MDT0001 complete [ 6878.497678] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 12:44:10 (1787589850) [ 6885.176920] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6885.827351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6885.832582] Lustre: Skipped 6 previous similar messages [ 6890.824874] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6900.977214] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.146306] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6913.090858] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6918.583871] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6926.472798] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6934.604782] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6940.711613] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6945.964730] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6946.265515] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6950.808990] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6951.049410] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6958.695767] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6958.921353] Lustre: lustre-OST0001: new disk, initializing [ 6958.925143] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6958.930549] Lustre: 221440:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6966.579475] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 6968.477714] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6968.496891] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6968.701727] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6976.997454] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6982.901925] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6983.256812] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6988.624015] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6988.999750] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6994.408479] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6994.424081] LustreError: Skipped 1 previous similar message [ 6994.432470] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6994.453453] Lustre: Skipped 17 previous similar messages [ 6997.622678] Lustre: server umount lustre-OST0001 complete [ 7004.374204] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7004.617847] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7009.878378] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7010.027936] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7039.931988] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7045.622126] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7045.639021] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7045.640995] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7045.661325] Lustre: Skipped 1 previous similar message [ 7047.736484] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7055.496107] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7057.187273] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7065.440186] Lustre: server umount lustre-OST0001 complete [ 7097.312219] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7097.617844] Lustre: server umount lustre-OST0000 complete [ 7106.017395] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7106.041870] Lustre: Skipped 9 previous similar messages [ 7109.161773] Lustre: server umount lustre-MDT0000 complete [ 7113.557144] LustreError: 218050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590087 with bad export cookie 251429156524315532 [ 7113.583293] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7113.586117] LustreError: 218050:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7120.090571] Lustre: server umount lustre-MDT0001 complete [ 7146.229256] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 12:48:38 (1787590118) [ 7153.115712] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7153.418664] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7153.453045] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7158.808533] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7167.828828] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7167.907412] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7167.936954] Lustre: 226686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7172.381183] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7179.720736] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7185.039755] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7192.060403] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7192.367234] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7198.658908] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7207.719107] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7216.575263] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7216.632032] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7224.062443] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7232.031722] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7233.951216] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7240.115568] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7240.377668] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7246.523314] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7246.814828] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7253.497527] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa0970b5af000.ost_server_uuid 50 [ 7255.315976] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa0970b5af000.ost_server_uuid in IDLE state after 0 sec [ 7256.342499] Lustre: Permanently deactivating lustre-OST0001 [ 7256.364141] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7260.144649] Lustre: Permanently reactivating lustre-OST0001 [ 7265.251633] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7265.263529] LustreError: Skipped 6 previous similar messages [ 7271.069792] Lustre: server umount lustre-OST0001 complete [ 7274.480261] LustreError: 228529:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7274.501475] LustreError: 228529:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 42 previous similar messages [ 7280.967602] Lustre: server umount lustre-OST0000 complete [ 7291.215983] Lustre: server umount lustre-MDT0000 complete [ 7295.523306] LustreError: 229556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590269 with bad export cookie 251429156524316820 [ 7295.526461] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7295.541442] LustreError: 229556:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7296.242755] Lustre: server umount lustre-MDT0001 complete [ 7320.393602] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 12:51:33 (1787590293) [ 7328.415773] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7328.619905] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7328.650193] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7328.655914] Lustre: Skipped 5 previous similar messages [ 7332.769479] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7341.984309] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7342.166089] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7342.200935] Lustre: 233382:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7348.489462] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7358.583856] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7364.486310] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7371.409963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7371.574545] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7371.595649] Lustre: Permanently deactivating lustre-OST0000 [ 7371.622148] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7378.636247] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7381.559611] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7387.001475] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7394.128172] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7394.192442] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7401.665598] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7403.828178] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7409.818260] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7411.588831] Lustre: Permanently reactivating lustre-OST0000 [ 7411.602639] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7411.610513] Lustre: Skipped 2 previous similar messages [ 7420.010952] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7420.012774] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 38 seconds. I think it's dead, and I am evicting it. exp 00000000c5b7150d, cur 1787590394 deadline 1787590386 last 1787590356 [ 7420.019666] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7420.050479] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7420.129164] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7429.051048] Lustre: server umount lustre-OST0001 complete [ 7439.090548] Lustre: server umount lustre-OST0000 complete [ 7457.248165] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7457.584729] Lustre: server umount lustre-MDT0000 complete [ 7461.666649] LustreError: 233418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590436 with bad export cookie 251429156524318493 [ 7461.677592] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7461.686712] LustreError: 233418:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7462.067575] Lustre: server umount lustre-MDT0001 complete [ 7474.878282] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7478.175923] Key type lgssc unregistered [ 7478.477984] LNet: 238269:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7478.484884] LNetError: 238269:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7478.510069] LNet: Removed LNI 192.168.201.103@tcp [ 7479.554148] Key type .llcrypt unregistered [ 7479.564961] Key type ._llcrypt unregistered [ 7505.238116] Key type ._llcrypt registered [ 7505.242547] Key type .llcrypt registered [ 7505.529568] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7506.861550] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7506.992205] alg: No test for adler32 (adler32-zlib) [ 7508.096502] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 7508.406560] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7510.160750] Key type lgssc registered [ 7511.358961] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7521.178927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7521.200756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7522.775319] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7527.701092] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7533.046388] LustreError: 239756:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7533.056317] LustreError: 239756:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7535.854776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7535.983089] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7536.013246] Lustre: 239745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7536.021293] Lustre: Permanently deactivating lustre-MDT0001 [ 7536.033291] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7536.432594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7537.296095] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7537.303360] LustreError: 239786:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7537.321268] LustreError: 239786:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7537.335271] LustreError: 239786:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 14, retries 1, failed: rc = -108 [ 7540.749503] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7547.896987] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7553.944517] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7561.193177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7561.531045] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7567.658575] LustreError: 241600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7567.675777] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7568.052571] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7572.965883] LustreError: 241600:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7575.956685] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7578.090695] LustreError: 241599:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7581.173206] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7583.406067] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7583.662638] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7588.843352] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7588.863933] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7590.938283] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7599.518537] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7603.730749] Lustre: Permanently reactivating lustre-MDT0001 [ 7603.741854] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7603.746956] Lustre: Skipped 2 previous similar messages [ 7609.327518] Lustre: lustre-MDT0001: Client 6caf81a9-9330-4f41-bfc6-acb25f12fb32 (at 192.168.201.3@tcp) reconnecting [ 7614.829986] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7616.287197] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7621.892786] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7622.110886] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7625.860422] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7626.127739] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7628.479677] Lustre: Permanently deactivating lustre-MDT0001 [ 7628.496565] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7628.500814] Lustre: Skipped 2 previous similar messages [ 7638.646029] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7649.258288] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7649.273636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7649.285237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7650.278244] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7650.278768] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7650.289419] Lustre: Skipped 2 previous similar messages [ 7650.311219] Lustre: Skipped 3 previous similar messages [ 7654.068234] Lustre: server umount lustre-MDT0000 complete [ 7655.400318] LustreError: 240500:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7655.424255] LustreError: 240500:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 7658.189367] LustreError: 239736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590632 with bad export cookie 3624584385713171234 [ 7658.201047] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7658.216819] LustreError: 239736:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7658.513703] Lustre: server umount lustre-MDT0001 complete [ 7664.854728] Lustre: server umount lustre-OST0000 complete [ 7677.257903] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:57:29 (1787590649) [ 7687.843831] Lustre: server umount lustre-OST0001 complete [ 7696.051398] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 7707.769612] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7746.998244] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7747.221302] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7747.227604] Lustre: Skipped 2 previous similar messages [ 7747.256860] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7747.364212] Lustre: lustre-MDT0000: new disk, initializing [ 7747.475339] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7747.505252] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7751.320568] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7761.005200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7761.136755] Lustre: 248065:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7761.214315] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7761.216894] Lustre: Skipped 1 previous similar message [ 7761.275847] Lustre: lustre-MDT0001: new disk, initializing [ 7761.384248] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7761.431586] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7761.437205] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7765.205340] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7772.694670] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7777.994670] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7786.892556] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7787.156747] Lustre: lustre-OST0000: new disk, initializing [ 7787.162143] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7787.167296] Lustre: 250033:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7787.259955] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7794.752575] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7794.759529] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7794.934253] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7794.976293] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7804.798489] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7806.949210] LustreError: 248074:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7816.102783] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7816.381822] Lustre: lustre-OST0001: new disk, initializing [ 7816.394765] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7816.416608] Lustre: 251151:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7817.024198] LustreError: 248074:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7817.088751] LustreError: 248072:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7823.492347] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7823.508821] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7823.692356] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7823.780832] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7827.114857] LustreError: 248072:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7833.993796] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7835.620654] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7839.202712] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7839.203193] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7839.211660] LustreError: Skipped 1 previous similar message [ 7839.220669] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7839.253428] Lustre: Skipped 1 previous similar message [ 7839.286055] Lustre: Skipped 1 previous similar message [ 7843.829895] Lustre: server umount lustre-OST0001 complete [ 7844.330043] LustreError: 250017:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7844.365884] LustreError: 250017:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7864.810836] LustreError: 250018:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7864.824836] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7864.839349] LustreError: 250018:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 7864.864997] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7864.885202] Lustre: Skipped 1 previous similar message [ 7868.506637] Lustre: server umount lustre-OST0000 complete [ 7874.018571] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7874.027618] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7874.040756] Lustre: Skipped 1 previous similar message [ 7874.045525] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7878.748172] Lustre: server umount lustre-MDT0000 complete [ 7882.956101] LustreError: 248057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787590857 with bad export cookie 3624584385713174048 [ 7882.965488] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7882.973484] LustreError: 248057:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7889.533377] Lustre: server umount lustre-MDT0001 complete [ 7901.213634] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 7903.872236] Key type lgssc unregistered [ 7904.180799] LNet: 253199:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7904.185995] LNetError: 253199:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7904.201707] LNet: Removed LNI 192.168.201.103@tcp [ 7904.863131] Key type .llcrypt unregistered [ 7904.866049] Key type ._llcrypt unregistered [ 7953.444462] Key type ._llcrypt registered [ 7953.447499] Key type .llcrypt registered [ 7953.612763] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 7955.537542] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7955.568089] alg: No test for adler32 (adler32-zlib) [ 7956.686369] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 7957.061196] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 7958.776174] Key type lgssc registered [ 7960.127407] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7969.557899] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7969.573764] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7970.925493] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7970.967067] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7971.180360] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7976.377311] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7985.181656] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7985.276793] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7985.306356] Lustre: 255409:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7985.511720] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7989.680652] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 7996.653265] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8005.007861] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8013.748706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8013.868941] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8014.212965] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8020.090305] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8020.289521] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8026.485737] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8081.378372] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8081.393679] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8081.403345] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8081.899715] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8081.906168] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8081.922323] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8083.339898] Lustre: server umount lustre-OST0000 complete [ 8088.206773] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8101.391279] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8101.718038] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8102.762397] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8102.768167] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8102.772649] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8102.787227] Lustre: Skipped 1 previous similar message [ 8107.454509] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8113.781754] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8154.098711] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8154.100106] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8154.104491] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8154.114688] LustreError: Skipped 1 previous similar message [ 8154.146506] Lustre: Skipped 1 previous similar message [ 8158.074576] Lustre: server umount lustre-OST0000 complete [ 8164.324923] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8164.326737] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8164.335058] Lustre: Skipped 1 previous similar message [ 8164.339993] Lustre: Skipped 1 previous similar message [ 8168.458033] Lustre: server umount lustre-MDT0000 complete [ 8172.500398] LustreError: 256150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591146 with bad export cookie 6672634588948010855 [ 8172.516415] LustreError: 256150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8172.516505] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8172.785420] Lustre: server umount lustre-MDT0001 complete [ 8186.973864] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8190.059853] Key type lgssc unregistered [ 8190.515676] LNet: 260837:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8190.533401] LNetError: 260837:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8190.579073] LNet: Removed LNI 192.168.201.103@tcp [ 8191.634065] Key type .llcrypt unregistered [ 8191.638960] Key type ._llcrypt unregistered [ 8216.729830] Key type ._llcrypt registered [ 8216.730931] Key type .llcrypt registered [ 8216.843118] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8218.027330] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8218.210061] alg: No test for adler32 (adler32-zlib) [ 8219.594865] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 8220.021687] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8221.736179] Key type lgssc registered [ 8223.193701] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8231.001355] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8231.017116] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8232.523813] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8238.093139] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8241.643667] LustreError: 262253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8241.677348] LustreError: 262253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8246.582507] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8246.753854] LustreError: 262254:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8246.938328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8252.003489] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8262.565746] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8269.569733] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8277.286864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8277.699266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8285.392557] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8293.930471] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8302.285077] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8333.282368] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8333.291153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8333.315762] Lustre: Skipped 1 previous similar message [ 8333.330220] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8336.830996] Lustre: server umount lustre-OST0000 complete [ 8343.526142] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8343.535509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8343.543443] Lustre: Skipped 1 previous similar message [ 8343.569413] Lustre: Skipped 2 previous similar messages [ 8346.819928] Lustre: server umount lustre-MDT0000 complete [ 8348.650979] LustreError: 262253:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8348.672076] LustreError: 262253:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8350.652475] LustreError: 262233:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591325 with bad export cookie 7777509403110580058 [ 8350.666242] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8351.009375] Lustre: server umount lustre-MDT0001 complete [ 8362.935526] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8366.436870] Key type lgssc unregistered [ 8366.736076] LNet: 266587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8366.740669] LNetError: 266587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8366.776134] LNet: Removed LNI 192.168.201.103@tcp [ 8368.075361] Key type .llcrypt unregistered [ 8368.077139] Key type ._llcrypt unregistered [ 8389.044210] Key type ._llcrypt registered [ 8389.045898] Key type .llcrypt registered [ 8389.201448] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8390.402635] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8390.536142] alg: No test for adler32 (adler32-zlib) [ 8391.849376] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 8392.276603] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8394.024443] Key type lgssc registered [ 8395.703242] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8404.934527] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8405.000494] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8406.520976] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8412.143289] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8415.727895] LustreError: 267813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8415.759279] LustreError: 267813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8420.837564] LustreError: 267814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8421.612821] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8422.049142] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8426.682345] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8435.207051] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8441.461888] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8448.409293] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8449.004267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8451.105592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8457.258453] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8467.413340] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8481.762564] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8481.771101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8481.795399] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8483.816557] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8483.834098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8483.847985] Lustre: Skipped 1 previous similar message [ 8485.853091] Lustre: server umount lustre-OST0000 complete [ 8493.537284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8493.548352] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8493.564437] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8499.170274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8499.202274] Lustre: Skipped 3 previous similar messages [ 8504.290211] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8504.293162] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8504.300799] Lustre: Skipped 1 previous similar message [ 8504.468506] Lustre: server umount lustre-MDT0000 complete [ 8508.228979] LustreError: 267794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591482 with bad export cookie 1410246969864702023 [ 8508.239250] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8508.427630] Lustre: server umount lustre-MDT0001 complete [ 8522.767854] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8527.609421] Key type lgssc unregistered [ 8527.934141] LNet: 271497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8527.953028] LNetError: 271497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8527.967180] LNet: Removed LNI 192.168.201.103@tcp [ 8529.135980] Key type .llcrypt unregistered [ 8529.138420] Key type ._llcrypt unregistered [ 8557.254508] Key type ._llcrypt registered [ 8557.261238] Key type .llcrypt registered [ 8557.414778] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8559.134722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8559.193906] alg: No test for adler32 (adler32-zlib) [ 8560.460704] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 8560.800637] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8562.496795] Key type lgssc registered [ 8564.024037] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8572.958859] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8572.980424] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8574.585576] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8580.101515] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8584.184198] LustreError: 272916:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8584.227457] LustreError: 272916:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8589.283120] LustreError: 272915:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8590.867250] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8591.446638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8599.406216] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8609.101541] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8616.440876] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8625.433914] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8626.059206] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8633.983918] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8641.511466] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8644.473920] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8672.225488] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8672.238802] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8672.262406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8675.303388] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8675.307413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8675.349669] Lustre: Skipped 1 previous similar message [ 8677.841624] Lustre: server umount lustre-OST0000 complete [ 8683.496404] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8683.506520] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8683.527725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8685.540705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8685.543493] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8688.045059] Lustre: server umount lustre-MDT0000 complete [ 8690.657102] LustreError: 273643:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8690.689451] LustreError: 273643:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8692.027976] LustreError: 272898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591666 with bad export cookie 12737520655146023285 [ 8692.042155] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8692.048391] LustreError: 272898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8692.568315] Lustre: server umount lustre-MDT0001 complete [ 8705.168285] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8708.726724] Key type lgssc unregistered [ 8709.071273] LNet: 277227:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8709.078617] LNetError: 277227:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8709.111346] LNet: Removed LNI 192.168.201.103@tcp [ 8710.045967] Key type .llcrypt unregistered [ 8710.048424] Key type ._llcrypt unregistered [ 8730.646703] Key type ._llcrypt registered [ 8730.649670] Key type .llcrypt registered [ 8730.760686] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8732.391216] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8732.475156] alg: No test for adler32 (adler32-zlib) [ 8733.554422] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 8733.885809] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8735.656211] Key type lgssc registered [ 8736.979634] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8745.871884] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8745.898522] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8747.345332] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8752.906476] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8756.729254] LustreError: 278463:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8756.755426] LustreError: 278463:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8761.864581] LustreError: 278465:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8764.080350] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8764.525366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8769.702601] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8778.185093] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8784.962945] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8792.422438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8792.569695] LustreError: 280312:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8792.974691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8798.231609] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 8801.310447] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8809.928471] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8823.778090] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8823.791654] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8823.800065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8824.821368] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8824.823921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8824.832461] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8827.746499] Lustre: server umount lustre-OST0000 complete [ 8835.044050] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8835.056404] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8835.061943] Lustre: Skipped 1 previous similar message [ 8835.079993] Lustre: Skipped 2 previous similar messages [ 8837.955909] Lustre: server umount lustre-MDT0000 complete [ 8840.166357] LustreError: 278473:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8840.193375] LustreError: 278473:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8842.184470] LustreError: 281415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787591816 with bad export cookie 5854706922797349954 [ 8842.196889] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8842.200753] LustreError: 281415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8842.681019] Lustre: server umount lustre-MDT0001 complete [ 8853.962988] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 8857.062459] Key type lgssc unregistered [ 8857.424186] LNet: 282169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8857.436190] LNetError: 282169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8857.453198] LNet: Removed LNI 192.168.201.103@tcp [ 8858.637253] Key type .llcrypt unregistered [ 8858.640668] Key type ._llcrypt unregistered [ 8871.640852] Key type ._llcrypt registered [ 8871.642889] Key type .llcrypt registered [ 8871.799989] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8891.548914] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 8907.873872] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 8909.223059] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8909.363786] alg: No test for adler32 (adler32-zlib) [ 8910.660142] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 8911.062777] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 8912.876497] Key type lgssc registered [ 8914.237554] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8955.233972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8955.252202] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8956.575790] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8956.618790] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8956.702123] Lustre: lustre-MDT0000: new disk, initializing [ 8956.790356] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8956.805341] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8962.727789] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8974.745971] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8974.875775] Lustre: 285725:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8974.940542] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8974.948435] Lustre: Skipped 1 previous similar message [ 8975.045508] Lustre: lustre-MDT0001: new disk, initializing [ 8975.184441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8975.247686] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8975.264292] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8979.700742] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 8989.815540] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8996.695575] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9007.242320] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9007.789978] Lustre: lustre-OST0000: new disk, initializing [ 9007.806665] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9007.829951] Lustre: 287695:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9008.060974] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9016.369781] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9017.982035] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9017.991865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9018.188189] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9027.438547] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9032.461258] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9032.668737] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9037.784037] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9038.056408] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9043.937452] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9043.938563] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9043.951269] LustreError: Skipped 1 previous similar message [ 9043.994498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9044.005460] Lustre: Skipped 1 previous similar message [ 9046.546726] Lustre: server umount lustre-OST0000 complete [ 9052.129789] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9052.137333] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9052.155075] Lustre: Skipped 1 previous similar message [ 9052.175316] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9054.177302] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9054.182579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9054.197132] Lustre: Skipped 1 previous similar message [ 9057.674856] Lustre: server umount lustre-MDT0000 complete [ 9061.991741] LustreError: 286521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592036 with bad export cookie 14008906169533575401 [ 9061.996134] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9062.011487] LustreError: 286521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9062.374706] Lustre: server umount lustre-MDT0001 complete [ 9070.615862] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 13:20:43 (1787592043) [ 9077.732766] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9078.248360] LustreError: 290108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9078.266214] LustreError: 290108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9078.324456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9082.034640] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9083.372315] LustreError: 290109:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9088.484948] LustreError: 290108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9092.372619] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9092.967629] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9100.226822] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9110.254146] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9115.727936] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9123.585669] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9124.005952] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9131.353560] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9139.920704] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9159.137539] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9159.146579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9159.173510] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9164.265724] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9164.281232] Lustre: Skipped 1 previous similar message [ 9165.219249] Lustre: server umount lustre-OST0000 complete [ 9169.383055] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9169.398755] Lustre: Skipped 1 previous similar message [ 9174.498412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9174.501116] Lustre: Skipped 4 previous similar messages [ 9175.580109] Lustre: server umount lustre-MDT0000 complete [ 9179.480366] LustreError: 291952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592153 with bad export cookie 14008906169533576276 [ 9179.481984] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9179.485768] LustreError: 291952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9179.852541] Lustre: server umount lustre-MDT0001 complete [ 9194.792760] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 9198.432877] Key type lgssc unregistered [ 9198.731754] LNet: 293673:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9198.737521] LNetError: 293673:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9199.784775] LNet: Removed LNI 192.168.201.103@tcp [ 9201.109572] Key type .llcrypt unregistered [ 9201.125954] Key type ._llcrypt unregistered [ 9233.954949] Key type ._llcrypt registered [ 9233.959079] Key type .llcrypt registered [ 9234.103348] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 9251.879414] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9253.586585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9253.670176] alg: No test for adler32 (adler32-zlib) [ 9254.765441] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 9255.217577] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 9256.953697] Key type lgssc registered [ 9258.406170] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9307.464813] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9307.478831] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9308.804342] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9308.891278] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9309.069597] Lustre: lustre-MDT0000: new disk, initializing [ 9309.389604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9309.429546] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9314.279851] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9324.463440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9324.551852] Lustre: 297499:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9324.575939] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9324.577970] Lustre: Skipped 1 previous similar message [ 9324.618720] Lustre: lustre-MDT0001: new disk, initializing [ 9324.683334] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9324.722496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9324.748185] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9328.733458] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9339.453079] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9348.123613] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9357.863896] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9358.204402] Lustre: lustre-OST2710: new disk, initializing [ 9358.206367] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9358.210207] Lustre: 299470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9358.342725] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9364.162979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9364.207046] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9364.465971] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9367.752394] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9379.070634] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9387.516524] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9387.726094] Lustre: lustre-OST03e8: new disk, initializing [ 9387.730492] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9387.738449] Lustre: 300535:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9387.881759] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9393.808449] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9394.946170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9394.950149] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9395.234834] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9403.669804] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9413.697392] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9414.258560] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9421.503606] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9421.811562] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9428.783299] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9429.157473] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9432.714438] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9432.938812] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9437.682540] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9437.697713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9437.721641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9441.252708] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9441.253414] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9441.262174] Lustre: Skipped 1 previous similar message [ 9441.301403] Lustre: Skipped 2 previous similar messages [ 9443.169248] Lustre: server umount lustre-MDT0000 complete [ 9446.836922] LustreError: 297492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592421 with bad export cookie 5701836839443564712 [ 9446.843543] LustreError: 297492:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9446.848842] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9447.177455] Lustre: server umount lustre-MDT0001 complete [ 9460.952160] Lustre: server umount lustre-OST2710 complete [ 9475.634981] Lustre: server umount lustre-OST03e8 complete [ 9497.893444] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [ 9511.000553] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9548.779659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9548.951721] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9548.977093] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9549.015654] Lustre: lustre-MDT0000: new disk, initializing [ 9549.059418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9549.073565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9553.250054] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9564.326199] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9564.441372] Lustre: 305790:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9564.484917] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9564.488288] Lustre: Skipped 1 previous similar message [ 9564.603377] Lustre: lustre-MDT0001: new disk, initializing [ 9564.736229] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9564.780883] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9564.801573] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9569.448867] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9578.388979] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9583.666171] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9590.078823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9590.409768] Lustre: lustre-OST0000: new disk, initializing [ 9590.416697] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9590.421933] Lustre: 307757:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9590.520687] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9596.488219] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9596.504518] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9596.641076] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9596.796772] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9606.693166] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9612.368322] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9612.762526] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9617.633436] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9617.990592] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9622.498744] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9622.500812] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9622.524819] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9622.538949] Lustre: Skipped 3 previous similar messages [ 9626.702425] Lustre: server umount lustre-OST0000 complete [ 9631.202732] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9631.216313] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9631.239943] Lustre: Skipped 1 previous similar message [ 9631.260308] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9637.198790] Lustre: server umount lustre-MDT0000 complete [ 9637.862389] LustreError: 305797:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9637.894596] LustreError: 305797:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9641.928191] LustreError: 305782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592616 with bad export cookie 5701836839443566392 [ 9641.932922] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9641.945919] LustreError: 305782:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9642.446971] Lustre: server umount lustre-MDT0001 complete [ 9651.758980] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 13:30:24 (1787592624) [ 9653.472307] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9655.011508] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 13:30:27 (1787592627) [ 9659.181419] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9685.310571] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9685.606164] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9685.633459] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9685.761517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9691.214966] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9700.046635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9700.112278] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9700.142542] Lustre: 311617:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9704.851423] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9726.599672] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9727.046482] Lustre: Denying initial registration attempt from nid 192.168.201.103@tcp, specified as failover [ 9727.063573] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 9727.084222] LustreError: 313070:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 9727.095744] LustreError: 313070:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 9727.103785] LustreError: 313070:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 9727.219696] Lustre: server umount lustre-OST0000 complete [ 9727.221820] LustreError: 313070:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 9736.162975] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9736.177888] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9736.191631] Lustre: Skipped 1 previous similar message [ 9736.198883] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9736.204044] Lustre: Skipped 2 previous similar messages [ 9741.678881] Lustre: server umount lustre-MDT0000 complete [ 9744.354987] LustreError: 311624:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9744.376992] LustreError: 311624:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9746.006837] LustreError: 311607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592720 with bad export cookie 5701836839443567274 [ 9746.012851] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9746.024058] LustreError: 311607:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9746.359225] Lustre: server umount lustre-MDT0001 complete [ 9759.006710] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 9761.280489] Key type lgssc unregistered [ 9761.584488] LNet: 314194:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9761.597182] LNetError: 314194:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9761.628073] LNet: Removed LNI 192.168.201.103@tcp [ 9762.752583] Key type .llcrypt unregistered [ 9762.754690] Key type ._llcrypt unregistered [ 9776.445361] Key type ._llcrypt registered [ 9776.448456] Key type .llcrypt registered [ 9776.639581] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9778.150687] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9778.314683] alg: No test for adler32 (adler32-zlib) [ 9779.506342] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 9779.704751] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [ 9781.368196] Key type lgssc registered [ 9782.766753] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9833.199599] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9842.701627] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9842.739671] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9844.109562] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9844.141082] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9844.311947] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9849.873825] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9861.883488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9861.977495] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9862.034383] Lustre: 316961:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9862.420522] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9867.612940] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9876.936548] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9884.467374] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9905.712446] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9905.963842] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9906.210418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9912.222809] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [ 9919.281711] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9924.080057] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9924.102806] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9927.758877] Lustre: server umount lustre-OST0000 complete [ 9934.305752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9934.306683] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9934.330394] Lustre: Skipped 1 previous similar message [ 9934.346639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9934.374199] Lustre: Skipped 2 previous similar messages [ 9939.721150] Lustre: server umount lustre-MDT0000 complete [ 9944.558088] LustreError: 318801:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9944.577963] LustreError: 318801:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9944.661728] LustreError: 316952:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787592919 with bad export cookie 3117698945316868010 [ 9944.681537] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9945.043673] Lustre: server umount lustre-MDT0001 complete [ 9960.349373] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [ 9965.343253] Key type lgssc unregistered [ 9965.764757] LNet: 320539:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9965.768265] LNetError: 320539:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9965.797710] LNet: Removed LNI 192.168.201.103@tcp [ 9967.366742] Key type .llcrypt unregistered [ 9967.369612] Key type ._llcrypt unregistered [ 9996.390587] Key type ._llcrypt registered [ 9996.392401] Key type .llcrypt registered [ 9996.484088] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [ 9997.482594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9997.511836] alg: No test for adler32 (adler32-zlib) [ 9998.859544] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [ 9999.227450] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [10000.939188] Key type lgssc registered [10002.120492] Lustre: Echo OBD driver; http://www.lustre.org/ [10012.205711] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10012.242727] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10013.783281] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10019.217154] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10023.911238] LustreError: 321958:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10023.943137] LustreError: 321958:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10029.029330] LustreError: 321959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10032.448148] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10033.034928] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10038.451735] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10047.066941] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10053.268503] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10062.054606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10062.604752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10070.856827] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10081.741604] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10099.153669] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.3@tcp (stopping) [10099.689242] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10099.689291] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10099.697806] Lustre: Skipped 1 previous similar message [10099.699408] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10099.699413] Lustre: Skipped 1 previous similar message [10102.593347] Lustre: server umount lustre-MDT0000 complete [10104.245484] LustreError: 323705:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10105.777477] LustreError: 321938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787593080 with bad export cookie 3731249960270603437 [10105.778873] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10105.794143] LustreError: 321938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10109.357842] LustreError: 321955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10109.374719] LustreError: 321955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10109.382608] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [10109.396550] Lustre: Skipped 1 previous similar message [10109.923197] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10109.940167] Lustre: Skipped 1 previous similar message [10114.475442] LustreError: 324451:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10114.475932] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [10114.497789] LustreError: 324451:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10114.513457] Lustre: Skipped 2 previous similar messages [10119.593266] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [10119.602368] Lustre: Skipped 1 previous similar message [10120.165954] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10120.812781] Lustre: server umount lustre-MDT0001 complete [10130.024696] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10138.967582] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10150.881511] LustreError: 321267:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000029d9b21e x1874427089299712/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10151.409666] LustreError: 323791:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10151.444431] LustreError: 323791:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10151.547941] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10151.614712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10157.850087] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10167.582756] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10167.836197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10170.928850] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10172.875680] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10181.070347] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10182.757907] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10189.442907] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10191.219282] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10198.500167] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10198.516177] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10198.539820] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10201.572856] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10203.795353] Lustre: server umount lustre-OST0000 complete [10208.739546] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10208.748422] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10214.671129] Lustre: server umount lustre-MDT0000 complete [10216.938813] LustreError: 325579:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10216.954282] LustreError: 325579:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10218.605118] LustreError: 321938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787593193 with bad export cookie 3731249960270610920 [10218.611395] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10218.613684] LustreError: 321938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10218.965730] Lustre: server umount lustre-MDT0001 complete [10230.718443] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [10233.692748] Key type lgssc unregistered [10234.077419] LNet: 328255:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10234.083059] LNetError: 328255:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10234.116864] LNet: Removed LNI 192.168.201.103@tcp [10235.175602] Key type .llcrypt unregistered [10235.179094] Key type ._llcrypt unregistered [10260.944674] Key type ._llcrypt registered [10260.946358] Key type .llcrypt registered [10261.055545] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [10262.336954] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10262.377414] alg: No test for adler32 (adler32-zlib) [10263.433429] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [10263.681291] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [10265.329108] Key type lgssc registered [10266.600670] Lustre: Echo OBD driver; http://www.lustre.org/ [10273.864025] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10273.896346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10275.371435] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10279.761871] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10285.543527] LustreError: 329674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10285.587882] LustreError: 329674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10288.286237] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10288.682244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10293.614964] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10300.856387] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10307.528978] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10310.114458] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10310.123019] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10310.126221] Lustre: Skipped 1 previous similar message [10314.213203] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10314.216114] Lustre: Skipped 1 previous similar message [10316.434163] Lustre: server umount lustre-MDT0000 complete [10319.329698] LustreError: 329674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10319.374989] LustreError: 329674:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10320.385124] LustreError: 329654:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787593294 with bad export cookie 3175562353190889328 [10320.402653] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10320.765103] Lustre: server umount lustre-MDT0001 complete [10335.993076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10336.494728] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10336.558485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10336.854914] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10342.830845] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10352.643056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10352.890309] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10352.984490] Lustre: 332267:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10353.710205] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10359.557586] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10367.657914] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10373.320880] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10382.698328] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10383.032683] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10383.058380] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10383.068076] LustreError: 334150:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [10383.076223] LustreError: 334150:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [10383.086056] LustreError: 334150:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10383.215355] Lustre: server umount lustre-OST0000 complete [10383.219378] LustreError: 334150:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10393.737542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10393.934752] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10394.155689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10399.865874] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10400.464125] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10409.062507] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10418.527615] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10418.608519] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10418.692503] Lustre: lustre-OST0001: new disk, initializing [10418.695770] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10418.701177] Lustre: 335694:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10418.824827] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10425.955270] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10425.975764] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10426.125551] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10426.931510] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10435.375659] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10441.704687] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10441.710134] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10441.750313] Lustre: Skipped 1 previous similar message [10441.777060] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10444.871665] Lustre: server umount lustre-OST0001 complete [10446.819623] LustreError: 334951:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10446.829061] LustreError: 334951:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10450.914693] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10450.922756] LustreError: Skipped 1 previous similar message [10450.926761] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10450.935963] Lustre: Skipped 1 previous similar message [10450.962603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10450.966976] Lustre: Skipped 1 previous similar message [10451.939353] LustreError: 334617:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10451.960204] LustreError: 334617:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10454.349214] Lustre: server umount lustre-OST0000 complete [10461.152964] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10461.163025] LustreError: Skipped 1 previous similar message [10461.167965] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10461.185392] Lustre: Skipped 1 previous similar message [10461.197243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10461.203057] Lustre: Skipped 2 previous similar messages [10465.154244] Lustre: server umount lustre-MDT0000 complete [10467.307978] LustreError: 332277:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10467.326230] LustreError: 332277:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10470.549193] LustreError: 333005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787593444 with bad export cookie 3175562353190889881 [10470.560677] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10470.566490] LustreError: 333005:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [10471.212216] Lustre: server umount lustre-MDT0001 complete [10487.986330] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [10490.934816] Key type lgssc unregistered [10491.184059] LNet: 337655:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10491.193717] LNetError: 337655:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10491.209177] LNet: Removed LNI 192.168.201.103@tcp [10492.637697] Key type .llcrypt unregistered [10492.657449] Key type ._llcrypt unregistered [10566.441066] Key type ._llcrypt registered [10566.446708] Key type .llcrypt registered [10566.540972] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [10582.420699] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [10584.208807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10584.270286] alg: No test for adler32 (adler32-zlib) [10585.224420] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [10585.501380] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [10587.185626] Key type lgssc registered [10588.726601] Lustre: Echo OBD driver; http://www.lustre.org/ [10629.059259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10629.116721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10630.510956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10630.558316] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10630.651373] Lustre: lustre-MDT0000: new disk, initializing [10630.767962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10630.819826] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10636.094798] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10648.355429] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10648.541745] Lustre: 342588:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10648.597543] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10648.601218] Lustre: Skipped 1 previous similar message [10648.703422] Lustre: lustre-MDT0001: new disk, initializing [10648.816487] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10648.858989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10648.876940] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10653.514670] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10663.649805] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10671.319508] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10679.207201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10679.394192] Lustre: lustre-OST0000: new disk, initializing [10679.396703] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10679.400501] Lustre: 344561:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10679.470798] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10686.060350] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10688.070852] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10688.078792] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10688.188826] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10694.643247] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10699.283697] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10699.587984] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10704.914903] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10705.101533] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10708.968140] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10708.973226] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10708.982094] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10708.993724] Lustre: Skipped 1 previous similar message [10709.010353] Lustre: Skipped 1 previous similar message [10713.247754] Lustre: server umount lustre-OST0000 complete [10719.204207] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10719.226677] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10724.162485] Lustre: server umount lustre-MDT0000 complete [10728.615966] LustreError: 343385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787593703 with bad export cookie 6509346349249991414 [10728.623556] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10728.628933] LustreError: 343385:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10729.029559] Lustre: server umount lustre-MDT0001 complete [10738.390686] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 13:48:30 (1787593710) [10747.243884] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 13:48:39 (1787593719) [10759.959815] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10760.813620] LustreError: 347314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10760.825168] LustreError: 347314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10760.865387] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10766.314147] LustreError: 347315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10766.746289] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10771.434262] LustreError: 347314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10776.550437] LustreError: 347315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10776.856780] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10777.381166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10782.868578] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10791.590296] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10798.378315] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10806.658880] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10807.223731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10816.581834] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10825.633125] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10839.884530] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [10842.402944] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [10844.740409] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [10846.657688] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [10848.552912] Lustre: Failing over lustre-MDT0000 [10848.768618] Lustre: server umount lustre-MDT0000 complete [10849.249326] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10849.270941] LustreError: Skipped 1 previous similar message [10849.286509] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10849.314296] Lustre: Skipped 1 previous similar message [10849.323921] LustreError: 347315:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10851.314398] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10851.349495] Lustre: Skipped 1 previous similar message [10857.905136] LustreError: 349804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10857.929470] LustreError: 349804:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10866.664909] Lustre: 340814:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787593825/real 1787593825] req@00000000877433a4 x1874427703797120/t0(0) o400->MGC192.168.201.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787593841 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10866.708983] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10868.756368] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10876.897470] LustreError: 340810:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000ebdfd836 x1874427703801856/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10876.936329] LustreError: 347314:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10876.952844] LustreError: 347314:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 14 previous similar messages [10877.208575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10877.253641] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10878.384805] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10882.543683] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10882.561077] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10882.622704] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10882.914543] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10893.054728] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10894.707718] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10898.602282] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [10900.942108] Lustre: Failing over lustre-MDT0000 [10901.414225] Lustre: server umount lustre-MDT0000 complete [10903.017593] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10903.024141] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10903.044673] Lustre: Skipped 2 previous similar messages [10909.098223] LustreError: 347311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10909.118572] LustreError: 347311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [10919.392211] Lustre: 340811:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787593877/real 1787593877] req@000000009e8bccf7 x1874427703818624/t0(0) o400->MGC192.168.201.103@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1787593893 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10919.408435] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10920.765378] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10928.611506] LustreError: 340810:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000006f4e5c30 x1874427703824256/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10928.930456] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10928.970872] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [10930.612937] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [10933.773335] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [10934.272234] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [10934.277651] Lustre: Skipped 2 previous similar messages [10934.307742] Lustre: lustre-MDT0000: Recovery over after 0:04, of 2 clients 2 recovered and 0 were evicted. [10934.344974] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10943.298479] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10944.813852] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10954.722919] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10954.726953] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10954.729137] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10954.729144] Lustre: Skipped 1 previous similar message [10954.749678] Lustre: Skipped 1 previous similar message [10956.136599] Lustre: server umount lustre-MDT0000 complete [10960.282090] LustreError: 347296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787593934 with bad export cookie 6509346349249994123 [10960.289240] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10960.292456] LustreError: 347296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10960.527687] Lustre: server umount lustre-MDT0001 complete [10975.961490] Lustre: server umount lustre-OST0000 complete [10993.000143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10993.789037] LustreError: 353678:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10993.812852] LustreError: 353678:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 22 previous similar messages [10993.927780] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10999.459102] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11008.094436] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11012.918268] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11020.282794] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11025.494395] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11032.516586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11039.028332] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11039.653363] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11046.792108] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11053.824438] Lustre: server umount lustre-OST0000 complete [11059.681586] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11059.685467] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11059.693546] Lustre: Skipped 2 previous similar messages [11059.707893] Lustre: Skipped 3 previous similar messages [11064.806712] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11064.819885] Lustre: Skipped 1 previous similar message [11066.983681] Lustre: server umount lustre-MDT0000 complete [11069.728271] Lustre: 340812:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787594028/real 1787594028] req@000000003b48400f x1874427703868416/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787594044 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11071.720159] LustreError: 353660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594046 with bad export cookie 6509346349249994872 [11071.732572] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11071.738710] LustreError: 353660:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11072.320274] Lustre: server umount lustre-MDT0001 complete [11084.850601] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [11087.717325] Key type lgssc unregistered [11088.153450] LNet: 357215:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11088.165268] LNetError: 357215:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11088.183898] LNet: Removed LNI 192.168.201.103@tcp [11089.236100] Key type .llcrypt unregistered [11089.238365] Key type ._llcrypt unregistered [11114.738846] Key type ._llcrypt registered [11114.741624] Key type .llcrypt registered [11114.876639] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [11115.266783] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11115.333860] alg: No test for adler32 (adler32-zlib) [11116.684693] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [11116.888094] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [11118.584266] Key type lgssc registered [11120.063647] Lustre: Echo OBD driver; http://www.lustre.org/ [11129.759398] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11129.796062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11131.292491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11136.397839] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11141.623492] LustreError: 358732:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11141.674541] LustreError: 358732:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11146.727369] LustreError: 358731:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11147.723370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11148.176726] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11154.535385] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11163.526431] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11170.048149] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11176.695404] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11176.866632] LustreError: 360551:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11177.046183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11182.098269] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11182.982642] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11189.888879] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11203.036962] Lustre: server umount lustre-MDT0000 complete [11203.558872] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11203.569169] LustreError: 358723:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11203.575890] Lustre: Skipped 2 previous similar messages [11203.589112] LustreError: 358723:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11206.729320] LustreError: 358712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594181 with bad export cookie 13766734275099963467 [11206.734416] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11206.737466] LustreError: 358712:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11207.174247] Lustre: server umount lustre-MDT0001 complete [11221.518456] Lustre: server umount lustre-OST0000 complete [11234.801896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11235.551931] LustreError: 362444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11235.646946] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11241.623316] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11245.049242] LustreError: 362445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11245.086378] LustreError: 362445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11251.532988] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11252.090427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11256.773704] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11265.080981] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11271.225199] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11278.048226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11278.427061] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11280.519997] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11284.716809] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11292.660780] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11303.397334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11303.410246] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11303.443247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11307.491329] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11307.491785] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11307.508762] Lustre: Skipped 2 previous similar messages [11309.426322] Lustre: server umount lustre-MDT0000 complete [11312.614889] LustreError: 362445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11312.626472] LustreError: 362445:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11313.657482] LustreError: 362424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594288 with bad export cookie 13766734275099964573 [11313.679851] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11313.680693] LustreError: 362424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11314.195396] Lustre: server umount lustre-MDT0001 complete [11328.740941] Lustre: server umount lustre-OST0000 complete [11346.106650] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11346.517857] LustreError: 366353:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11346.582536] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11351.552236] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11360.302658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11366.168595] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11374.278303] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11381.259822] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11389.425321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11389.970766] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11389.980823] Lustre: Skipped 1 previous similar message [11390.971537] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11397.618674] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11406.744592] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11415.521771] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11415.527429] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11415.542761] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11415.561058] Lustre: Skipped 2 previous similar messages [11419.032937] Lustre: server umount lustre-OST0000 complete [11425.768268] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11425.774270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11425.789764] Lustre: Skipped 1 previous similar message [11425.804542] Lustre: Skipped 2 previous similar messages [11429.569281] Lustre: server umount lustre-MDT0000 complete [11430.893937] LustreError: 366350:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11430.933878] LustreError: 366350:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11433.458034] LustreError: 366333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594407 with bad export cookie 13766734275099965602 [11433.476502] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11433.501965] LustreError: 366333:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [11433.807642] Lustre: server umount lustre-MDT0001 complete [11445.745628] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [11449.669538] Key type lgssc unregistered [11450.013815] LNet: 369887:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11450.018792] LNetError: 369887:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11450.054400] LNet: Removed LNI 192.168.201.103@tcp [11451.008767] Key type .llcrypt unregistered [11451.010119] Key type ._llcrypt unregistered [11483.731108] Key type ._llcrypt registered [11483.738419] Key type .llcrypt registered [11483.911328] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [11485.603865] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11485.681600] alg: No test for adler32 (adler32-zlib) [11486.929904] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [11487.240542] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [11489.056398] Key type lgssc registered [11490.142563] Lustre: Echo OBD driver; http://www.lustre.org/ [11498.862989] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11498.878286] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11498.883962] LustreError: 371344:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11498.929074] LustreError: 371344:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11498.940921] LustreError: 371344:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11498.952740] LustreError: 371344:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11498.963878] LustreError: 371344:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11507.131662] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11507.136721] LustreError: 371742:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11507.151664] LustreError: 371742:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11507.157545] LustreError: 371742:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11507.162934] LustreError: 371742:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11507.180779] LustreError: 371742:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11530.663652] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [11534.574812] Key type lgssc unregistered [11534.892139] LNet: 372749:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11534.899931] LNetError: 372749:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11534.917686] LNet: Removed LNI 192.168.201.103@tcp [11535.497386] Key type .llcrypt unregistered [11535.499424] Key type ._llcrypt unregistered [11558.407762] Key type ._llcrypt registered [11558.409544] Key type .llcrypt registered [11558.564111] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [11575.772179] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [11577.657867] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11577.690108] alg: No test for adler32 (adler32-zlib) [11578.766749] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [11579.039362] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [11580.745080] Key type lgssc registered [11582.055653] Lustre: Echo OBD driver; http://www.lustre.org/ [11621.218256] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11621.248531] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11622.578030] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11622.624592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11622.754724] Lustre: lustre-MDT0000: new disk, initializing [11622.845925] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11622.873700] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11628.716573] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11640.712448] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11640.830285] Lustre: 375911:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11640.881977] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11640.885970] Lustre: Skipped 1 previous similar message [11640.944193] Lustre: lustre-MDT0001: new disk, initializing [11641.024101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11641.111684] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11641.128458] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11646.577816] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11658.921102] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11666.404819] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11674.643468] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11674.917783] Lustre: lustre-OST0000: new disk, initializing [11674.922982] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11674.929445] Lustre: 377878:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11675.023774] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11682.663214] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11682.899918] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11682.910212] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11683.162686] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11693.192127] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11699.139666] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11699.364499] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11705.200827] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11705.569837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11708.908643] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11708.909263] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11708.923910] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11708.931986] LustreError: Skipped 1 previous similar message [11714.024960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11714.034252] Lustre: Skipped 2 previous similar messages [11719.139046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11719.152062] Lustre: Skipped 1 previous similar message [11722.209170] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11722.502579] Lustre: server umount lustre-OST0000 complete [11728.354185] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11728.362225] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11728.382160] Lustre: Skipped 1 previous similar message [11728.388130] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11729.382796] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11733.339747] Lustre: server umount lustre-MDT0000 complete [11738.135994] LustreError: 375902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594712 with bad export cookie 1235773864698695314 [11738.149769] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11738.668258] Lustre: server umount lustre-MDT0001 complete [11747.662460] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [11749.608987] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 14:05:22 (1787594722) [11756.217730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11756.686099] LustreError: 380339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11756.704315] LustreError: 380339:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11756.747655] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11762.150797] LustreError: 380340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11762.746536] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11767.272045] LustreError: 380339:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11773.415170] LustreError: 380340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11773.454464] LustreError: 380340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11775.659353] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11776.451771] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11781.992093] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11792.226622] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11800.256535] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11808.316187] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11809.134701] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11817.811853] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11828.395386] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11837.009460] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11837.245681] Lustre: lustre-OST0001: new disk, initializing [11837.251500] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [11837.259625] Lustre: 383179:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11837.410659] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [11846.100984] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [11846.370823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [11846.383912] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [11846.527108] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [11857.190651] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11861.986492] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11861.994732] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11862.005453] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [11862.013907] Lustre: Skipped 1 previous similar message [11862.018302] Lustre: Skipped 3 previous similar messages [11867.527148] Lustre: server umount lustre-OST0001 complete [11869.101924] LustreError: 382160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11877.365891] LustreError: 383555:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11877.394109] LustreError: 383555:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11887.594813] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11887.597121] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11887.622391] Lustre: Skipped 1 previous similar message [11887.625307] Lustre: Skipped 3 previous similar messages [11897.838368] LustreError: 382160:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11897.887139] LustreError: 382160:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [11900.384290] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11900.469325] Lustre: server umount lustre-OST0000 complete [11908.068100] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11908.076543] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11908.081261] Lustre: Skipped 6 previous similar messages [11919.328163] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [11919.763043] Lustre: server umount lustre-MDT0000 complete [11924.288992] LustreError: 380320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787594898 with bad export cookie 1235773864698696196 [11924.294179] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11924.303792] LustreError: 380320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [11930.834068] Lustre: server umount lustre-MDT0001 complete [11943.091289] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [11946.776359] Key type lgssc unregistered [11947.081077] LNet: 385173:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11947.096096] LNetError: 385173:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11947.121850] LNet: Removed LNI 192.168.201.103@tcp [11948.123564] Key type .llcrypt unregistered [11948.130536] Key type ._llcrypt unregistered [12011.227081] Key type ._llcrypt registered [12011.228870] Key type .llcrypt registered [12011.359612] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12013.044949] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12013.141612] alg: No test for adler32 (adler32-zlib) [12014.254727] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [12014.481254] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12016.152208] Key type lgssc registered [12017.712975] Lustre: Echo OBD driver; http://www.lustre.org/ [12026.609246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12026.643761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12028.022767] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12028.044393] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12028.294529] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12033.442925] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12043.264649] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12043.431426] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12043.498332] Lustre: 387975:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12044.010743] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12049.524392] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12058.391405] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12064.931793] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12069.870284] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12069.890882] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12069.927418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12070.883344] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12070.884791] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12073.480258] Lustre: server umount lustre-MDT0000 complete [12077.738515] LustreError: 387966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595052 with bad export cookie 17528271720774762707 [12077.745160] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12077.746344] LustreError: 387966:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12078.338994] Lustre: server umount lustre-MDT0001 complete [12088.873160] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 14:11:01 (1787595061) [12095.633251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12096.104406] LustreError: 390487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12096.115173] LustreError: 390487:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12096.150744] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12101.146774] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12101.602102] LustreError: 390488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12106.730822] LustreError: 390487:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12109.775561] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12110.264889] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12115.479656] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12123.962914] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12130.246660] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12138.303238] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12138.508330] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12138.759425] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12146.562770] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12155.950710] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12163.357319] Lustre: Permanently deactivating lustre-OST0000 [12163.381696] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12172.322726] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12174.884866] LustreError: 393526:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12174.899794] LustreError: 393526:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12174.909130] LustreError: 393526:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12178.486595] LustreError: 393575:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12178.494795] LustreError: 393575:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12186.922323] Lustre: server umount lustre-OST0000 complete [12189.897411] LustreError: 393825:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12189.907400] LustreError: 393825:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12192.228932] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12192.240951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12192.267943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12192.278718] Lustre: Skipped 1 previous similar message [12197.864355] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12197.875554] Lustre: Skipped 2 previous similar messages [12202.263611] Lustre: server umount lustre-MDT0000 complete [12202.979264] LustreError: 390483:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12203.000462] LustreError: 390483:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12206.154491] LustreError: 393183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595180 with bad export cookie 17528271720774763288 [12206.156992] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12206.178391] LustreError: 393183:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12216.096885] Lustre: server umount lustre-MDT0001 complete [12226.700767] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12232.428864] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12235.448085] LustreError: 394985:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12242.662449] Lustre: 395324:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.103@tcp [12244.995777] Lustre: server umount MGS complete [12256.598376] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12256.906531] LustreError: 396048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12256.913844] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12256.920046] Lustre: Skipped 1 previous similar message [12256.956550] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12262.866740] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12267.492261] LustreError: 396049:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12267.524655] LustreError: 396049:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12273.595240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12274.041476] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12274.144982] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12280.084903] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12288.493972] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12294.228948] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12296.605229] Lustre: Permanently reactivating lustre-OST0000 [12296.621926] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12296.626104] Lustre: Skipped 2 previous similar messages [12323.296562] LustreError: 397782:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12331.678736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12332.197742] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12339.052404] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12347.385513] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12354.018158] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12354.026067] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12354.031986] Lustre: Skipped 1 previous similar message [12354.036075] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12354.039675] Lustre: Skipped 1 previous similar message [12354.691646] Lustre: server umount lustre-OST0000 complete [12359.140453] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12359.153971] Lustre: Skipped 2 previous similar messages [12364.258305] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12364.265585] Lustre: Skipped 3 previous similar messages [12365.096714] Lustre: server umount lustre-MDT0000 complete [12368.766346] LustreError: 396028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595343 with bad export cookie 17528271720774764338 [12368.770393] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12368.779795] LustreError: 396028:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12369.046412] Lustre: server umount lustre-MDT0001 complete [12379.198147] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12382.595802] Key type lgssc unregistered [12383.199074] LNet: 400367:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12383.225231] LNetError: 400367:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12383.265054] LNet: Removed LNI 192.168.201.103@tcp [12384.279545] Key type .llcrypt unregistered [12384.284561] Key type ._llcrypt unregistered [12405.175996] Key type ._llcrypt registered [12405.178820] Key type .llcrypt registered [12405.329861] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [12423.083576] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12424.834342] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12424.952157] alg: No test for adler32 (adler32-zlib) [12426.201484] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [12426.568875] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12428.296233] Key type lgssc registered [12429.881389] Lustre: Echo OBD driver; http://www.lustre.org/ [12471.407144] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12471.489420] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12472.831485] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12472.898166] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12473.004892] Lustre: lustre-MDT0000: new disk, initializing [12473.078086] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12473.100398] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12478.485505] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12490.644241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12490.769619] Lustre: 403525:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12490.845294] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12490.849277] Lustre: Skipped 1 previous similar message [12490.966986] Lustre: lustre-MDT0001: new disk, initializing [12491.113494] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12491.182979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12491.206561] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12496.408805] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12508.046433] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12514.904979] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12522.838988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12523.126783] Lustre: lustre-OST0000: new disk, initializing [12523.130380] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12523.137233] Lustre: 405495:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12523.201743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12529.273267] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12529.299082] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12529.466729] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12530.353576] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12541.236867] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12547.254398] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12547.548507] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12552.589337] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12552.962813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12560.366315] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12560.381594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12560.390115] Lustre: Skipped 1 previous similar message [12560.406602] Lustre: Skipped 1 previous similar message [12561.780582] Lustre: server umount lustre-OST0000 complete [12568.043360] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12568.049527] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12568.062450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12570.598703] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12570.603524] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12570.618838] Lustre: Skipped 1 previous similar message [12575.715363] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12575.719089] Lustre: Skipped 1 previous similar message [12579.822729] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12579.825856] Lustre: Skipped 1 previous similar message [12580.321703] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12580.729726] Lustre: server umount lustre-MDT0000 complete [12584.806372] LustreError: 403517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595559 with bad export cookie 8845311596940451206 [12584.820295] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12584.822887] LustreError: 403517:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [12585.185959] Lustre: server umount lustre-MDT0001 complete [12595.154417] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [12597.288487] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 14:19:29 (1787595569) [12604.137405] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12605.050736] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12610.535836] LustreError: 407956:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12610.561101] LustreError: 407956:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12610.653503] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12614.635060] LustreError: 407957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12618.660140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12618.919148] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12623.648416] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12633.076522] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12639.870624] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12647.822963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12648.318938] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12655.312588] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12664.353525] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12666.540544] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [12669.478207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [12674.021871] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12674.029269] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12674.036493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12675.395074] Lustre: server umount lustre-OST0000 complete [12680.681132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12680.686662] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12680.699789] Lustre: Skipped 1 previous similar message [12686.682742] Lustre: server umount lustre-MDT0000 complete [12689.379988] LustreError: 410729:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12689.455348] LustreError: 410729:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12692.525340] LustreError: 407939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595666 with bad export cookie 8845311596940452081 [12692.526897] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12692.535112] LustreError: 407939:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12692.927168] Lustre: server umount lustre-MDT0001 complete [12707.789560] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12711.356445] Key type lgssc unregistered [12711.682207] LNet: 411685:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12711.685870] LNetError: 411685:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12711.712253] LNet: Removed LNI 192.168.201.103@tcp [12712.659585] Key type .llcrypt unregistered [12712.662779] Key type ._llcrypt unregistered [12740.527556] Key type ._llcrypt registered [12740.528920] Key type .llcrypt registered [12740.613362] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12740.876356] Key type .llcrypt unregistered [12740.890742] Key type ._llcrypt unregistered [12762.618918] Key type ._llcrypt registered [12762.624990] Key type .llcrypt registered [12762.755137] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12764.052543] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12764.142832] alg: No test for adler32 (adler32-zlib) [12765.327733] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [12765.608652] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12767.408431] Key type lgssc registered [12768.774423] Lustre: Echo OBD driver; http://www.lustre.org/ [12777.719804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12777.756793] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12779.441492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12784.127920] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12788.708266] LustreError: 413906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12788.729706] LustreError: 413906:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12793.224276] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12793.604934] LustreError: 413907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12793.638028] LustreError: 413907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [12793.652597] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12798.948531] LustreError: 413906:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12801.414551] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12804.068194] LustreError: 413902:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12804.098465] LustreError: 413902:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12805.102183] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [12808.832600] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12810.210395] LustreError: 413907:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12810.235816] LustreError: 413907:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12815.162394] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12815.677584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12820.141938] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12829.910944] Lustre: server umount lustre-OST0000 complete [12835.814133] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [12835.826847] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12835.847674] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12841.441706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12841.449605] Lustre: Skipped 3 previous similar messages [12843.552700] Lustre: server umount lustre-MDT0000 complete [12846.564632] LustreError: 415928:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12846.575205] LustreError: 415928:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [12846.810729] LustreError: 413885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787595821 with bad export cookie 794624282245383946 [12846.814831] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12846.821844] LustreError: 413885:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12857.625724] Lustre: server umount lustre-MDT0001 complete [12871.863267] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [12876.010104] Key type lgssc unregistered [12876.351312] LNet: 417056:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12876.369067] LNetError: 417056:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12876.387558] LNet: Removed LNI 192.168.201.103@tcp [12877.285425] Key type .llcrypt unregistered [12877.288358] Key type ._llcrypt unregistered [12904.220570] Key type ._llcrypt registered [12904.222208] Key type .llcrypt registered [12904.367605] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [12906.448128] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12906.470353] alg: No test for adler32 (adler32-zlib) [12907.487025] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [12907.705967] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [12909.349497] Key type lgssc registered [12910.732521] Lustre: Echo OBD driver; http://www.lustre.org/ [12919.810966] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12919.837080] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12935.456419] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [12940.576309] LustreError: 418449:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.103@tcp: failed processing log, type 4: rc = -110 [12967.200689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12974.740178] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12981.835754] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12988.067301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12988.369810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12988.428265] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [12992.986975] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [12993.511382] LustreError: 418475:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12993.541662] LustreError: 418475:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12998.634197] LustreError: 418474:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13001.707220] LustreError: 419488:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13001.721107] LustreError: 419488:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13003.314680] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13003.802387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13009.711452] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13019.026453] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13025.410281] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13034.343845] Lustre: server umount lustre-OST0000 complete [13038.382943] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13038.390206] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13038.423846] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13039.588036] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13039.591877] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13039.626862] Lustre: Skipped 1 previous similar message [13044.705823] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13047.909948] Lustre: server umount lustre-MDT0000 complete [13049.832967] Lustre: 417785:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787596008/real 1787596008] req@00000000fcdc6829 x1874430139595008/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1787596024 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13049.842682] LustreError: 419492:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13049.863866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13049.898547] LustreError: 419492:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13050.529404] Lustre: 417788:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787596008/real 1787596008] req@0000000057b3f4c9 x1874430139595264/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1787596024 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13051.700396] LustreError: 418457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787596026 with bad export cookie 14778400988807253440 [13051.703079] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13051.711300] LustreError: 418457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13052.178840] Lustre: server umount lustre-MDT0001 complete [13063.787945] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [13067.206882] Key type lgssc unregistered [13067.562629] LNet: 422026:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13067.568295] LNetError: 422026:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13067.614206] LNet: Removed LNI 192.168.201.103@tcp [13068.707371] Key type .llcrypt unregistered [13068.711989] Key type ._llcrypt unregistered [13097.194497] Key type ._llcrypt registered [13097.200691] Key type .llcrypt registered [13097.377469] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [13098.826878] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13098.991726] alg: No test for adler32 (adler32-zlib) [13100.115337] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [13100.433167] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [13102.120201] Key type lgssc registered [13103.371595] Lustre: Echo OBD driver; http://www.lustre.org/ [13112.045262] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13112.112049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13113.970189] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13119.554540] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13123.625883] LustreError: 423444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13123.657309] LustreError: 423444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13128.677993] LustreError: 423445:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13129.667118] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13130.158731] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13135.127965] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13143.675535] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13149.978673] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13157.218265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13157.461083] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13159.547640] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13165.014580] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13173.753969] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13180.387255] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13180.411981] Lustre: Skipped 1 previous similar message [13180.420359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13180.435745] Lustre: Skipped 2 previous similar messages [13181.466875] Lustre: server umount lustre-MDT0000 complete [13181.901098] LustreError: 423440:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.3@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13183.853396] LustreError: 423425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.3@tcp arrived at 1787596158 with bad export cookie 9090842168338518190 [13183.873496] LustreError: 423425:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13185.508539] LustreError: 423438:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13185.533317] LustreError: 423438:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13186.014789] LustreError: 423426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787596160 with bad export cookie 9090842168338517294 [13186.034314] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13186.036316] LustreError: 423426:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13190.625397] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13190.630319] LustreError: 423444:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13190.635171] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13190.635183] Lustre: Skipped 1 previous similar message [13190.635888] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13190.695870] LustreError: 423444:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13191.653438] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13192.347913] Lustre: server umount lustre-OST0000 complete [13199.321148] Lustre: server umount lustre-MDT0001 complete [13209.927554] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [13213.093673] Key type lgssc unregistered [13213.485493] LNet: 427132:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13213.495118] LNetError: 427132:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13213.533371] LNet: Removed LNI 192.168.201.103@tcp [13214.675383] Key type .llcrypt unregistered [13214.681974] Key type ._llcrypt unregistered [13242.628693] Key type ._llcrypt registered [13242.631954] Key type .llcrypt registered [13242.798366] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [13244.284874] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13244.514327] alg: No test for adler32 (adler32-zlib) [13245.815367] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [13246.092794] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [13247.752331] Key type lgssc registered [13249.193838] Lustre: Echo OBD driver; http://www.lustre.org/ [13257.734036] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13257.765429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13259.125492] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13264.424586] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13269.490399] LustreError: 428553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13269.521365] LustreError: 428553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13274.602626] LustreError: 428552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13274.630669] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13274.899995] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13280.039830] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13288.350135] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13293.277759] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13301.531493] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13302.236112] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13304.334752] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13309.304370] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13318.717798] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13324.783744] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13324.804196] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13325.767977] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [13325.772202] Lustre: Skipped 1 previous similar message [13328.878502] Lustre: server umount lustre-MDT0001 complete [13329.898508] LustreError: 428552:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13329.929582] LustreError: 428552:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13335.023131] LustreError: 428553:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13335.045818] LustreError: 428553:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13340.133418] LustreError: 429542:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13340.136316] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13340.141354] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13340.141364] Lustre: Skipped 1 previous similar message [13340.145413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13342.046146] Lustre: server umount lustre-OST0000 complete [13347.032471] Lustre: server umount lustre-MDT0000 complete [13362.546753] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [13365.888668] Key type lgssc unregistered [13366.257834] LNet: 432242:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13366.268112] LNetError: 432242:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13366.285033] LNet: Removed LNI 192.168.201.103@tcp [13367.122483] Key type .llcrypt unregistered [13367.126463] Key type ._llcrypt unregistered [13396.841421] Key type ._llcrypt registered [13396.843867] Key type .llcrypt registered [13396.952061] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [13414.194260] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [13415.482479] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13415.622373] alg: No test for adler32 (adler32-zlib) [13416.670276] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [13416.977711] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [13418.728296] Key type lgssc registered [13420.324210] Lustre: Echo OBD driver; http://www.lustre.org/ [13462.234224] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [13479.509226] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13479.534953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13480.870394] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13480.901644] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13480.974279] Lustre: lustre-MDT0000: new disk, initializing [13481.066890] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13481.082300] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13486.700549] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13504.681378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13505.127753] Lustre: lustre-OST0000: new disk, initializing [13505.133835] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13505.138928] Lustre: Skipped 1 previous similar message [13505.145583] Lustre: 437279:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13505.236792] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13511.817298] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13511.826433] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13511.841976] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13512.462360] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13521.971964] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13538.413737] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13538.552945] Lustre: 436260:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13538.613908] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13538.670881] Lustre: lustre-MDT0001: new disk, initializing [13538.820027] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13538.902836] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [13538.917486] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [13544.027975] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13554.831929] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [13555.029757] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [13561.046517] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [13561.345966] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [13568.961507] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.3@tcp (stopping) [13569.507381] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13569.550639] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13569.554346] Lustre: Skipped 1 previous similar message [13572.879459] Lustre: server umount lustre-MDT0001 complete [13579.760740] LustreError: 436272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13579.769224] LustreError: 436272:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [13584.875148] LustreError: 436267:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13584.913601] LustreError: 436267:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13590.021141] LustreError: 437588:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13590.060690] LustreError: 437588:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13593.057814] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13593.060343] LustreError: 436271:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13593.090895] Lustre: Skipped 1 previous similar message [13593.093550] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13597.540866] Lustre: server umount lustre-OST0000 complete [13598.178743] LustreError: 436272:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13608.851953] Lustre: server umount lustre-MDT0000 complete [13624.497835] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [13628.269567] Key type lgssc unregistered [13628.630723] LNet: 441030:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13628.638983] LNetError: 441030:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13628.654386] LNet: Removed LNI 192.168.201.103@tcp [13630.133593] Key type .llcrypt unregistered [13630.135322] Key type ._llcrypt unregistered [13675.511173] Key type ._llcrypt registered [13675.517619] Key type .llcrypt registered [13675.688016] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [13676.050610] Key type .llcrypt unregistered [13676.058544] Key type ._llcrypt unregistered [13692.589359] Key type ._llcrypt registered [13692.591212] Key type .llcrypt registered [13692.732807] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [13693.712722] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13693.758134] alg: No test for adler32 (adler32-zlib) [13694.856485] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [13695.122110] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [13696.856228] Key type lgssc registered [13698.812491] Lustre: Echo OBD driver; http://www.lustre.org/ [13729.726613] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13729.759757] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13731.165762] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13731.209621] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13731.360561] Lustre: lustre-MDT0000: new disk, initializing [13731.448847] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13731.477757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13735.907291] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13746.770913] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13746.942718] Lustre: 444412:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13747.043713] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13747.047281] Lustre: Skipped 1 previous similar message [13747.145394] Lustre: lustre-MDT0001: new disk, initializing [13747.228821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13747.273253] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13747.297810] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13752.605136] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13763.158754] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13769.398807] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13777.216641] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13777.449354] Lustre: lustre-OST0000: new disk, initializing [13777.451707] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13777.455327] Lustre: 446381:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13777.556518] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13784.133211] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13784.148888] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13784.301032] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13785.593042] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13796.390182] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13803.489582] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13803.508144] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13803.529707] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13805.028159] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13805.031941] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13805.048719] Lustre: Skipped 1 previous similar message [13805.061112] Lustre: Skipped 2 previous similar messages [13809.782923] Lustre: server umount lustre-MDT0000 complete [13814.501549] LustreError: 445209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787596788 with bad export cookie 6092431177913315010 [13814.511959] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13814.525329] LustreError: 445209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13814.997625] Lustre: server umount lustre-MDT0001 complete [13829.724461] Lustre: server umount lustre-OST0000 complete [13842.014060] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 14:40:14 (1787596814) [13850.348664] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13850.833541] LustreError: 448408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13850.861915] LustreError: 448408:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13850.942733] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13856.193659] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13856.239965] LustreError: 448409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13861.354024] LustreError: 448408:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13866.474996] LustreError: 448409:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13867.218761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13867.714735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13873.407626] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13882.293551] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13888.863729] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13909.490572] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13909.837229] Lustre: Found index 0 for lustre-OST0000, updating log [13910.166697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13915.422434] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [13922.449697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [13926.350897] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13938.150655] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13938.168045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13938.177858] Lustre: Skipped 1 previous similar message [13941.930178] Lustre: server umount lustre-OST0000 complete [13960.677390] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13960.682357] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13960.706683] Lustre: Skipped 1 previous similar message [13960.712960] Lustre: Skipped 4 previous similar messages [13964.844114] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13964.860323] Lustre: Skipped 1 previous similar message [13966.748451] Lustre: server umount lustre-MDT0000 complete [13970.515303] LustreError: 448389:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787596944 with bad export cookie 6092431177913316396 [13970.529737] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13970.837822] Lustre: server umount lustre-MDT0001 complete [13972.773758] LNet: 1 local NIs in recovery (showing 1): 192.168.201.103@tcp [13980.297714] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 14:42:32 (1787596952) [14608.778061] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14609.330450] LustreError: 454822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14609.425141] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14614.499221] LustreError: 454823:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14615.162317] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [14619.621378] LustreError: 454822:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14624.875540] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14625.277247] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14630.510034] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [14639.169160] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14645.968312] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14663.968279] LNet: 1 local NIs in recovery (showing 1): 192.168.201.103@tcp [14667.312763] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14667.803684] Lustre: Found index 0 for lustre-OST0000, updating log [14668.353702] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14676.016629] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [14684.692643] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14687.976694] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14688.002375] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14688.014656] Lustre: Skipped 2 previous similar messages [14693.270513] Lustre: server umount lustre-OST0000 complete [14698.530847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14698.533848] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14698.559957] Lustre: Skipped 1 previous similar message [14698.577355] Lustre: Skipped 1 previous similar message [14700.642960] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14700.652038] Lustre: Skipped 3 previous similar messages [14703.945849] Lustre: server umount lustre-MDT0000 complete [14704.805942] LustreError: 454817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14704.823024] LustreError: 454817:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [14708.304832] LustreError: 454804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787597682 with bad export cookie 6092431177913317488 [14708.317331] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14708.332345] LustreError: 454804:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 7 previous similar messages [14708.777647] Lustre: server umount lustre-MDT0001 complete [14724.448886] LNet: 1 local NIs in recovery (showing 1): 192.168.201.103@tcp [14742.407837] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [14754.462040] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [14795.857386] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 14:56:08 (1787597768) [14814.824360] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [14817.863898] Key type lgssc unregistered [14818.217776] LNet: 461443:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14818.221663] LNetError: 461443:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14931.879754] LNet: Removed LNI 192.168.201.103@tcp [14933.251981] Key type .llcrypt unregistered [14933.254643] Key type ._llcrypt unregistered [14951.048499] Key type ._llcrypt registered [14951.051729] Key type .llcrypt registered [14951.163761] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [14952.850570] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14952.946553] alg: No test for adler32 (adler32-zlib) [14954.032703] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [14954.356078] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [14956.048147] Key type lgssc registered [14957.839934] Lustre: Echo OBD driver; http://www.lustre.org/ [14963.334471] LNet: Added LNI 192.168.201.103@tcp100 [8/256/0/180] [14964.602158] LNet: Added LNI 192.168.201.103@tcp101 [8/256/0/180] [14965.534664] LNet: Added LNI 192.168.201.103@tcp102 [8/256/0/180] [14966.549680] LNet: Added LNI 192.168.201.103@tcp103 [8/256/0/180] [14967.713917] LNet: Added LNI 192.168.201.103@tcp104 [8/256/0/180] [14968.644312] LNet: Added LNI 192.168.201.103@tcp105 [8/256/0/180] [14969.688811] LNet: Added LNI 192.168.201.103@tcp106 [8/256/0/180] [14970.707438] LNet: Added LNI 192.168.201.103@tcp107 [8/256/0/180] [14971.674359] LNet: Added LNI 192.168.201.103@tcp108 [8/256/0/180] [14972.583524] LNet: Added LNI 192.168.201.103@tcp109 [8/256/0/180] [14973.564949] LNet: Added LNI 192.168.201.103@tcp110 [8/256/0/180] [14974.603605] LNet: Added LNI 192.168.201.103@tcp111 [8/256/0/180] [14975.551919] LNet: Added LNI 192.168.201.103@tcp112 [8/256/0/180] [14976.448168] LNet: Added LNI 192.168.201.103@tcp113 [8/256/0/180] [14977.531668] LNet: Added LNI 192.168.201.103@tcp114 [8/256/0/180] [14978.482494] LNet: Added LNI 192.168.201.103@tcp115 [8/256/0/180] [14979.574530] LNet: Added LNI 192.168.201.103@tcp116 [8/256/0/180] [14980.625427] LNet: Added LNI 192.168.201.103@tcp117 [8/256/0/180] [14981.642760] LNet: Added LNI 192.168.201.103@tcp118 [8/256/0/180] [14982.447870] LNet: Added LNI 192.168.201.103@tcp119 [8/256/0/180] [14983.433702] LNet: Added LNI 192.168.201.103@tcp120 [8/256/0/180] [14984.569105] LNet: Added LNI 192.168.201.103@tcp121 [8/256/0/180] [14985.756069] LNet: Added LNI 192.168.201.103@tcp122 [8/256/0/180] [14986.924935] LNet: Added LNI 192.168.201.103@tcp123 [8/256/0/180] [14988.103673] LNet: Added LNI 192.168.201.103@tcp124 [8/256/0/180] [14989.150196] LNet: Added LNI 192.168.201.103@tcp125 [8/256/0/180] [14990.401039] LNet: Added LNI 192.168.201.103@tcp126 [8/256/0/180] [14991.365586] LNet: Added LNI 192.168.201.103@tcp127 [8/256/0/180] [14992.513725] LNet: Added LNI 192.168.201.103@tcp128 [8/256/0/180] [14993.516407] LNet: Added LNI 192.168.201.103@tcp129 [8/256/0/180] [14994.452083] LNet: Added LNI 192.168.201.103@tcp130 [8/256/0/180] [14995.427580] LNet: Added LNI 192.168.201.103@tcp131 [8/256/0/180] [14996.319141] LNet: Added LNI 192.168.201.103@tcp132 [8/256/0/180] [14997.350927] LNet: Added LNI 192.168.201.103@tcp133 [8/256/0/180] [14998.446849] LNet: Added LNI 192.168.201.103@tcp134 [8/256/0/180] [14999.404480] LNet: Added LNI 192.168.201.103@tcp135 [8/256/0/180] [15007.470558] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15007.549145] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15008.695163] Lustre: 464476:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [15008.804387] LNetError: 464498:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [15008.819447] LNetError: Protocol error connecting to 192.168.201.103@tcp122 at host 192.168.201.103:988: is it running a compatible version of Lustre? [15008.842590] LNetError: Skipped 7 previous similar messages [15008.897703] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15008.917880] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15008.978646] Lustre: lustre-MDT0000: new disk, initializing [15009.066482] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15009.079496] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15009.697282] LNet: 5 local NIs in recovery (showing 5): 192.168.201.103@tcp124, 192.168.201.103@tcp128, 192.168.201.103@tcp129, 192.168.201.103@tcp127, 192.168.201.103@tcp122 [15009.700490] LNetError: 464497:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [15009.718723] LNetError: 464497:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 7 previous similar messages [15009.765660] LNetError: Protocol error connecting to 192.168.201.103@tcp100 at host 192.168.201.103:988: is it running a compatible version of Lustre? [15009.787394] LNetError: Skipped 3 previous similar messages [15014.681971] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15027.431376] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15027.653184] Lustre: 464493:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15027.781408] Lustre: 465298:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [15027.801756] Lustre: 465298:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15027.808646] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15027.822166] Lustre: Skipped 1 previous similar message [15027.836926] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [15027.933475] Lustre: lustre-MDT0001: new disk, initializing [15028.096562] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15028.169670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15028.190362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15033.340531] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15039.341081] LNetError: 464498:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [15039.364278] LNetError: 464498:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 5 previous similar messages [15039.382559] LNetError: Protocol error connecting to 192.168.201.103@tcp120 at host 192.168.201.103:988: is it running a compatible version of Lustre? [15042.405312] LNetError: 464504:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [15042.431485] LNetError: Protocol error connecting to 192.168.201.103@tcp128 at host 192.168.201.103:988: is it running a compatible version of Lustre? [15043.811938] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15046.497126] LNetError: 461935:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [15046.501605] LNetError: 461935:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 2 previous similar messages [15046.507754] LNetError: Protocol error connecting to 192.168.201.103@tcp124 at host 192.168.201.103:988: is it running a compatible version of Lustre? [15046.514944] LNetError: Skipped 2 previous similar messages [15049.981300] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15054.709641] LNetError: 464498:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [15054.726442] LNetError: 464498:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [15054.743704] LNetError: Protocol error connecting to 192.168.201.103@tcp128 at host 192.168.201.103:988: is it running a compatible version of Lustre? [15054.753509] LNetError: Skipped 4 previous similar messages [15057.632062] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15058.140100] Lustre: lustre-OST0000: new disk, initializing [15058.148905] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15058.169378] Lustre: 466470:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15058.306287] Lustre: 466475:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [15058.325436] Lustre: 466475:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [15058.391059] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15065.146494] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15066.680562] Lustre: 466884:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.103@tcp: can't add NID 192.168.201.103@tcp130: rc = -75 [15066.710457] Lustre: 466884:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15066.787779] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15066.797590] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15066.979508] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15072.111942] LNetError: 464498:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [15072.149946] LNetError: 464498:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 10 previous similar messages [15072.173198] LNetError: Protocol error connecting to 192.168.201.103@tcp124 at host 192.168.201.103:988: is it running a compatible version of Lustre? [15072.200158] LNetError: Skipped 10 previous similar messages [15075.670990] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15087.593368] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15087.597298] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15087.613300] LustreError: Skipped 1 previous similar message [15087.614124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15087.666345] Lustre: Skipped 1 previous similar message [15089.610628] Lustre: server umount lustre-OST0000 complete [15094.760295] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15094.773974] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15094.794590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15094.797233] Lustre: Skipped 1 previous similar message [15097.826868] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15097.828536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15102.945463] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15102.955590] Lustre: Skipped 2 previous similar messages [15106.917561] LNetError: 461936:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.103 [15106.930377] LNetError: 461936:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 12 previous similar messages [15106.944065] LNetError: Protocol error connecting to 192.168.201.103@tcp128 at host 192.168.201.103:988: is it running a compatible version of Lustre? [15106.954680] LNetError: Skipped 12 previous similar messages [15108.064181] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15108.395886] Lustre: server umount lustre-MDT0000 complete [15113.029398] LustreError: 465296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787598087 with bad export cookie 6515500694919316301 [15113.033823] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15113.045719] LustreError: 465296:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15113.848897] Lustre: server umount lustre-MDT0001 complete [15126.358248] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [15130.663812] Key type lgssc unregistered [15131.315845] LNet: 468400:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15131.330114] LNetError: 468400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15131.353879] LNet: Removed LNI 192.168.201.103@tcp [15132.384224] LNet: Removed LNI 192.168.201.103@tcp100 [15132.391360] LNetError: 468400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15132.408450] LNetError: 468400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15132.421960] LNet: Removed LNI 192.168.201.103@tcp101 [15133.478954] LNet: Removed LNI 192.168.201.103@tcp102 [15133.485179] LNetError: 468400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15133.509613] LNetError: 468400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15133.547306] LNet: Removed LNI 192.168.201.103@tcp103 [15133.579546] LNet: Removed LNI 192.168.201.103@tcp104 [15134.624445] LNet: Removed LNI 192.168.201.103@tcp105 [15135.648227] LNet: Removed LNI 192.168.201.103@tcp106 [15135.651473] LNetError: 468400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15135.659738] LNetError: 468400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 3 previous similar messages [15135.673612] LNet: Removed LNI 192.168.201.103@tcp107 [15136.736876] LNet: Removed LNI 192.168.201.103@tcp108 [15137.760283] LNet: Removed LNI 192.168.201.103@tcp109 [15138.784159] LNet: Removed LNI 192.168.201.103@tcp110 [15139.808188] LNet: Removed LNI 192.168.201.103@tcp111 [15139.809762] LNetError: 468400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15139.820983] LNetError: 468400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15139.853200] LNet: Removed LNI 192.168.201.103@tcp112 [15140.896213] LNet: Removed LNI 192.168.201.103@tcp113 [15141.920423] LNet: Removed LNI 192.168.201.103@tcp114 [15142.945591] LNet: Removed LNI 192.168.201.103@tcp115 [15143.968321] LNet: Removed LNI 192.168.201.103@tcp116 [15145.056310] LNet: Removed LNI 192.168.201.103@tcp117 [15146.082254] LNet: Removed LNI 192.168.201.103@tcp118 [15147.106985] LNet: Removed LNI 192.168.201.103@tcp119 [15148.128303] LNet: Removed LNI 192.168.201.103@tcp120 [15148.131287] LNetError: 468400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15148.155308] LNetError: 468400:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15148.162896] LNet: Removed LNI 192.168.201.103@tcp121 [15149.216338] LNet: Removed LNI 192.168.201.103@tcp122 [15150.240344] LNet: Removed LNI 192.168.201.103@tcp123 [15151.264186] LNet: Removed LNI 192.168.201.103@tcp124 [15152.288673] LNet: Removed LNI 192.168.201.103@tcp125 [15153.312693] LNet: Removed LNI 192.168.201.103@tcp126 [15154.337111] LNet: Removed LNI 192.168.201.103@tcp127 [15155.361051] LNet: Removed LNI 192.168.201.103@tcp128 [15156.384824] LNet: Removed LNI 192.168.201.103@tcp129 [15156.390456] LNet: Removed LNI 192.168.201.103@tcp130 [15156.398166] LNet: Removed LNI 192.168.201.103@tcp131 [15156.407237] LNet: Removed LNI 192.168.201.103@tcp132 [15156.438050] LNet: Removed LNI 192.168.201.103@tcp133 [15156.442194] LNet: Removed LNI 192.168.201.103@tcp134 [15156.462941] LNet: Removed LNI 192.168.201.103@tcp135 [15157.632972] Key type .llcrypt unregistered [15157.639759] Key type ._llcrypt unregistered [15184.113323] Key type ._llcrypt registered [15184.115794] Key type .llcrypt registered [15184.235331] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [15202.027845] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [15203.941273] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15203.994257] alg: No test for adler32 (adler32-zlib) [15205.038843] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [15205.272292] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [15206.947054] Key type lgssc registered [15208.021344] Lustre: Echo OBD driver; http://www.lustre.org/ [15246.608698] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 15:03:39 (1787598219) [15267.133594] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [15270.238697] Key type lgssc unregistered [15270.611455] LNet: 472497:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15270.622375] LNetError: 472497:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15270.638966] LNet: Removed LNI 192.168.201.103@tcp [15271.762476] Key type .llcrypt unregistered [15271.767205] Key type ._llcrypt unregistered [15290.321319] Key type ._llcrypt registered [15290.323359] Key type .llcrypt registered [15290.466338] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [15291.945183] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15291.984073] alg: No test for adler32 (adler32-zlib) [15292.990665] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [15293.185516] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [15294.840195] Key type lgssc registered [15295.998311] Lustre: Echo OBD driver; http://www.lustre.org/ [15305.412341] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15305.445691] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15306.821679] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15306.862853] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15306.989807] Lustre: lustre-MDT0000: new disk, initializing [15307.046084] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15307.067467] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15312.155975] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15324.429093] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15324.588606] Lustre: 473712:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15324.663212] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15324.669049] Lustre: Skipped 1 previous similar message [15324.728281] Lustre: lustre-MDT0001: new disk, initializing [15324.803762] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15324.842463] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15324.853918] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15330.264310] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15341.865421] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15348.768714] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15355.960484] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15356.208845] Lustre: lustre-OST0000: new disk, initializing [15356.215910] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15356.227940] Lustre: 475679:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15356.329713] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15363.173987] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15364.680113] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15364.697075] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15364.884363] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15372.582214] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15377.345868] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing lnet_if_list [15383.079799] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15383.103798] LNet: Added LNI 192.168.201.103@tcp10 [8/256/0/180] [15388.912698] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15388.937160] LNet: Added LNI 192.168.201.103@tcp11 [8/256/0/180] [15393.316598] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15393.338276] LNet: Added LNI 192.168.201.103@tcp12 [8/256/0/180] [15398.766764] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15398.800489] LNet: Added LNI 192.168.201.103@tcp13 [8/256/0/180] [15403.767996] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15403.808178] LNet: Added LNI 192.168.201.103@tcp14 [8/256/0/180] [15408.636192] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15408.664759] LNet: Added LNI 192.168.201.103@tcp15 [8/256/0/180] [15413.705457] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15413.756898] LNet: Added LNI 192.168.201.103@tcp16 [8/256/0/180] [15418.379466] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15418.410727] LNet: Added LNI 192.168.201.103@tcp17 [8/256/0/180] [15423.426705] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15423.461752] LNet: Added LNI 192.168.201.103@tcp18 [8/256/0/180] [15428.800973] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15428.818143] LNet: Added LNI 192.168.201.103@tcp19 [8/256/0/180] [15434.312777] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [15434.357158] LNet: Added LNI 192.168.201.103@tcp20 [8/256/0/180] [15439.997323] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [15440.029373] LNet: Added LNI 192.168.201.103@tcp21 [8/256/0/180] [15445.278599] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [15445.303571] LNet: Added LNI 192.168.201.103@tcp22 [8/256/0/180] [15451.182281] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [15451.201457] LNet: Added LNI 192.168.201.103@tcp23 [8/256/0/180] [15456.678867] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [15456.724677] LNet: Added LNI 192.168.201.103@tcp24 [8/256/0/180] [15463.217428] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [15463.237991] LNet: Added LNI 192.168.201.103@tcp25 [8/256/0/180] [15468.839364] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [15468.869448] LNet: Added LNI 192.168.201.103@tcp26 [8/256/0/180] [15473.857380] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [15473.881497] LNet: Added LNI 192.168.201.103@tcp27 [8/256/0/180] [15479.146512] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [15479.174488] LNet: Added LNI 192.168.201.103@tcp28 [8/256/0/180] [15485.048822] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [15485.075943] LNet: Added LNI 192.168.201.103@tcp29 [8/256/0/180] [15491.008914] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [15491.038325] LNet: Added LNI 192.168.201.103@tcp30 [8/256/0/180] [15497.606633] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [15497.655499] LNet: Added LNI 192.168.201.103@tcp31 [8/256/0/180] [15504.538343] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [15504.555296] LNet: Added LNI 192.168.201.103@tcp32 [8/256/0/180] [15511.237885] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [15511.275828] LNet: Added LNI 192.168.201.103@tcp33 [8/256/0/180] [15519.084617] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [15519.115540] LNet: Added LNI 192.168.201.103@tcp34 [8/256/0/180] [15526.092164] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [15526.110627] LNet: Added LNI 192.168.201.103@tcp35 [8/256/0/180] [15531.763807] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [15531.789192] LNet: Added LNI 192.168.201.103@tcp36 [8/256/0/180] [15537.621970] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [15537.691087] LNet: Added LNI 192.168.201.103@tcp37 [8/256/0/180] [15545.984714] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [15546.013918] LNet: Added LNI 192.168.201.103@tcp38 [8/256/0/180] [15553.894292] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [15553.928637] LNet: Added LNI 192.168.201.103@tcp39 [8/256/0/180] [15561.174681] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [15561.203246] LNet: Added LNI 192.168.201.103@tcp40 [8/256/0/180] [15571.698928] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [15571.746864] LNet: Added LNI 192.168.201.103@tcp41 [8/256/0/180] [15579.999760] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [15580.032990] LNet: Added LNI 192.168.201.103@tcp42 [8/256/0/180] [15587.460396] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [15587.488411] LNet: Added LNI 192.168.201.103@tcp43 [8/256/0/180] [15594.805497] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [15594.845923] LNet: Added LNI 192.168.201.103@tcp44 [8/256/0/180] [15603.429786] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [15603.459175] LNet: Added LNI 192.168.201.103@tcp45 [8/256/0/180] [15610.350852] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [15610.415974] LNet: Added LNI 192.168.201.103@tcp46 [8/256/0/180] [15619.097951] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [15619.191105] LNet: Added LNI 192.168.201.103@tcp47 [8/256/0/180] [15629.917192] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [15629.950251] LNet: Added LNI 192.168.201.103@tcp48 [8/256/0/180] [15641.444305] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [15641.465956] LNet: Added LNI 192.168.201.103@tcp49 [8/256/0/180] [15650.916486] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [15650.946423] LNet: Added LNI 192.168.201.103@tcp50 [8/256/0/180] [15672.297019] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15672.299825] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15672.305888] Lustre: Skipped 2 previous similar messages [15672.317635] Lustre: Skipped 2 previous similar messages [15677.410641] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15677.415014] Lustre: Skipped 2 previous similar messages [15677.961536] Lustre: server umount lustre-MDT0000 complete [15682.165213] LustreError: 473704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787598656 with bad export cookie 13183634351636879346 [15682.191221] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15682.206434] LustreError: 473704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15682.538816] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15683.070977] Lustre: server umount lustre-MDT0001 complete [15702.941532] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [15711.798732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15727.591636] LustreError: 472981:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000002cd8b851 x1874432640592128/t0(0) o250->MGC192.168.201.103@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [15728.080143] LustreError: 475673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15728.087499] LustreError: 475673:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [15728.183830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15733.219794] LustreError: 475674:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15733.804740] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15738.350447] LustreError: 475673:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15738.377699] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15743.035597] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15743.359840] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15747.398509] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15748.464513] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [15754.264133] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15759.382172] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15779.304355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15779.311719] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15779.311897] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15779.318862] Lustre: Skipped 1 previous similar message [15783.205155] Lustre: server umount lustre-OST0000 complete [15789.542871] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15789.543104] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15789.547153] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15789.547161] Lustre: Skipped 1 previous similar message [15789.556808] Lustre: Skipped 1 previous similar message [15793.883894] Lustre: server umount lustre-MDT0000 complete [15794.660937] LustreError: 499921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15794.684903] LustreError: 499921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [15797.311550] LustreError: 473703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787598771 with bad export cookie 13183634351636881915 [15797.317536] LustreError: 473703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [15797.318652] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15797.695934] Lustre: server umount lustre-MDT0001 complete [15809.504931] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [15812.561231] Key type lgssc unregistered [15812.805961] LNet: 502701:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15812.812476] LNetError: 502701:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15812.820512] LNet: Removed LNI 192.168.201.103@tcp [15812.822915] LNet: Removed LNI 192.168.201.103@tcp10 [15812.827993] LNet: Removed LNI 192.168.201.103@tcp11 [15812.832274] LNet: Removed LNI 192.168.201.103@tcp12 [15812.836498] LNet: Removed LNI 192.168.201.103@tcp13 [15812.839588] LNet: Removed LNI 192.168.201.103@tcp14 [15812.841845] LNet: Removed LNI 192.168.201.103@tcp15 [15812.844589] LNet: Removed LNI 192.168.201.103@tcp16 [15812.848576] LNet: Removed LNI 192.168.201.103@tcp17 [15812.851580] LNet: Removed LNI 192.168.201.103@tcp18 [15812.854431] LNet: Removed LNI 192.168.201.103@tcp19 [15812.857610] LNet: Removed LNI 192.168.201.103@tcp20 [15812.861869] LNet: Removed LNI 192.168.201.103@tcp21 [15812.865414] LNet: Removed LNI 192.168.201.103@tcp22 [15812.868319] LNet: Removed LNI 192.168.201.103@tcp23 [15812.873317] LNet: Removed LNI 192.168.201.103@tcp24 [15812.877316] LNet: Removed LNI 192.168.201.103@tcp25 [15812.879450] LNet: Removed LNI 192.168.201.103@tcp26 [15812.882537] LNet: Removed LNI 192.168.201.103@tcp27 [15812.886830] LNet: Removed LNI 192.168.201.103@tcp28 [15812.890797] LNet: Removed LNI 192.168.201.103@tcp29 [15812.893936] LNet: Removed LNI 192.168.201.103@tcp30 [15812.898658] LNet: Removed LNI 192.168.201.103@tcp31 [15812.902553] LNet: Removed LNI 192.168.201.103@tcp32 [15812.905995] LNet: Removed LNI 192.168.201.103@tcp33 [15812.909974] LNet: Removed LNI 192.168.201.103@tcp34 [15812.913670] LNet: Removed LNI 192.168.201.103@tcp35 [15812.917369] LNet: Removed LNI 192.168.201.103@tcp36 [15812.920993] LNet: Removed LNI 192.168.201.103@tcp37 [15812.924338] LNet: Removed LNI 192.168.201.103@tcp38 [15812.926476] LNet: Removed LNI 192.168.201.103@tcp39 [15812.929820] LNet: Removed LNI 192.168.201.103@tcp40 [15812.932753] LNet: Removed LNI 192.168.201.103@tcp41 [15812.934630] LNet: Removed LNI 192.168.201.103@tcp42 [15812.938336] LNet: Removed LNI 192.168.201.103@tcp43 [15812.941112] LNet: Removed LNI 192.168.201.103@tcp44 [15812.948826] LNet: Removed LNI 192.168.201.103@tcp45 [15812.954919] LNet: Removed LNI 192.168.201.103@tcp46 [15812.957781] LNet: Removed LNI 192.168.201.103@tcp47 [15812.961531] LNet: Removed LNI 192.168.201.103@tcp48 [15812.964651] LNet: Removed LNI 192.168.201.103@tcp49 [15812.979375] LNet: Removed LNI 192.168.201.103@tcp50 [15813.693802] Key type .llcrypt unregistered [15813.695461] Key type ._llcrypt unregistered [15835.537223] Key type ._llcrypt registered [15835.539683] Key type .llcrypt registered [15835.650646] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [15852.147461] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [15853.043717] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15853.152541] alg: No test for adler32 (adler32-zlib) [15854.375029] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [15854.779982] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [15856.481657] Key type lgssc registered [15857.892394] Lustre: Echo OBD driver; http://www.lustre.org/ [15898.148538] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15898.222285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15899.722479] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15899.772935] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15899.918307] Lustre: lustre-MDT0000: new disk, initializing [15900.062985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15900.086961] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15905.487413] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15917.434183] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15917.612262] Lustre: 505900:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15917.694822] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15917.703765] Lustre: Skipped 1 previous similar message [15917.789768] Lustre: lustre-MDT0001: new disk, initializing [15917.963429] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15918.023573] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15918.034807] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15923.762653] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15934.143440] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15940.425257] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15948.905335] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15949.195189] Lustre: *** cfs_fail_loc=90f, val=0*** [15949.278650] Lustre: lustre-OST0000: new disk, initializing [15949.282313] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15949.288284] Lustre: 507917:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15949.429629] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15955.058795] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15955.067300] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15955.222536] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15956.717816] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [15967.915927] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15981.031487] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15981.032848] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15981.045022] LustreError: Skipped 1 previous similar message [15981.045318] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15981.078945] Lustre: Skipped 1 previous similar message [15982.425321] Lustre: server umount lustre-OST0000 complete [15989.729741] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [15989.734045] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15989.749361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15989.755234] Lustre: Skipped 1 previous similar message [15991.268611] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15991.269564] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15995.442623] Lustre: server umount lustre-MDT0000 complete [15999.837852] LustreError: 505894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787598974 with bad export cookie 14572428537273611485 [15999.840936] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15999.851810] LustreError: 505894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16000.228798] Lustre: server umount lustre-MDT0001 complete [16013.438750] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [16016.494557] Key type lgssc unregistered [16016.925452] LNet: 509829:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16016.930701] LNetError: 509829:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16016.959100] LNet: Removed LNI 192.168.201.103@tcp [16018.134407] Key type .llcrypt unregistered [16018.140364] Key type ._llcrypt unregistered [16039.586890] Key type ._llcrypt registered [16039.589494] Key type .llcrypt registered [16039.692991] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [16041.105072] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16041.149452] alg: No test for adler32 (adler32-zlib) [16042.287835] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [16042.526328] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [16044.168249] Key type lgssc registered [16045.372775] Lustre: Echo OBD driver; http://www.lustre.org/ [16054.857746] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16054.906073] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16056.558828] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16061.051318] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [16066.027965] LustreError: 511105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16066.054347] LustreError: 511105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [16070.171674] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16070.642183] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16075.527502] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [16083.740194] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16090.004332] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16097.444702] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16097.715978] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16105.834544] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [16114.417255] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16126.034768] loop0: detected capacity change from 0 to 204800000 [16126.314482] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16126.329293] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16126.341388] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16133.954204] loop0: detected capacity change from 0 to 204800000 [16134.164568] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16134.187532] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16134.207765] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16140.551641] loop0: detected capacity change from 0 to 204800000 [16147.011194] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16147.164262] Lustre: fs15246-OST0000: new disk, initializing [16147.167697] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16147.172446] Lustre: 514696:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16147.220609] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16153.846560] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [16165.413761] loop1: detected capacity change from 0 to 204800000 [16172.732304] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16172.818199] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16172.853129] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16172.956745] Lustre: fs15246-MDT0000: new disk, initializing [16173.110388] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16173.134920] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16173.303662] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16174.378839] Lustre: 515819:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000003f509a42 x1874433425851264/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1787599164 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16179.469886] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [16179.498374] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16179.505025] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16179.566311] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16208.870978] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16208.889393] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16210.516683] Lustre: server umount fs15246-MDT0000 complete [16213.992090] LustreError: 511101:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16219.114549] LustreError: 511100:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16220.108414] Lustre: server umount fs15246-OST0000 complete [16232.932545] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16232.941213] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16232.952562] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16233.954982] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16233.967548] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16233.984074] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16239.592707] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16239.612029] Lustre: Skipped 3 previous similar messages [16244.707242] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16244.721984] Lustre: Skipped 1 previous similar message [16246.240313] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16246.493337] Lustre: server umount lustre-OST0000 complete [16254.946290] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16254.947220] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16254.954745] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16254.964440] Lustre: Skipped 1 previous similar message [16254.980105] Lustre: Skipped 1 previous similar message [16258.263866] Lustre: server umount lustre-MDT0000 complete [16260.069840] LustreError: 511105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16260.088700] LustreError: 511105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [16263.348201] LustreError: 511086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787599237 with bad export cookie 635711421599300262 [16263.350915] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16263.356972] LustreError: 511086:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [16263.719914] Lustre: server umount lustre-MDT0001 complete [16278.228042] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [16282.087731] Key type lgssc unregistered [16282.482875] LNet: 518921:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16282.491145] LNetError: 518921:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16282.514078] LNet: Removed LNI 192.168.201.103@tcp [16283.596397] Key type .llcrypt unregistered [16283.598527] Key type ._llcrypt unregistered [16306.463071] Key type ._llcrypt registered [16306.466477] Key type .llcrypt registered [16306.774115] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [16308.100565] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16308.283211] alg: No test for adler32 (adler32-zlib) [16309.461287] Lustre: Lustre: Build Version: 2.17.57_80_g8672ae7 [16309.732726] LNet: Added LNI 192.168.201.103@tcp [8/256/0/180] [16311.424187] Key type lgssc registered [16312.476830] Lustre: Echo OBD driver; http://www.lustre.org/ [16357.589279] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [16371.223924] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [16412.058577] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16412.081233] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16413.298356] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16413.326499] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16413.388276] Lustre: lustre-MDT0000: new disk, initializing [16413.515302] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16413.529911] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16419.754770] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [16429.756346] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16429.811062] Lustre: 523392:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16429.862799] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16429.866758] Lustre: Skipped 1 previous similar message [16429.915911] Lustre: lustre-MDT0001: new disk, initializing [16429.983914] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16430.023111] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16430.030446] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16434.442353] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [16444.993816] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16450.541279] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16457.771383] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16458.005685] Lustre: lustre-OST0000: new disk, initializing [16458.010283] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16458.017862] Lustre: 525359:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16458.117921] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16463.938739] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16463.948482] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16464.117953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16464.434902] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_default_debug -1 all [16474.428316] Lustre: DEBUG MARKER: oleg103-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16479.782901] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [16480.074413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [16485.262232] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [16485.608399] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [16489.963545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16489.979891] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16489.981779] LustreError: Skipped 1 previous similar message [16489.982991] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16490.019048] Lustre: Skipped 1 previous similar message [16493.965641] Lustre: server umount lustre-OST0000 complete [16500.193685] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16500.219323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16500.229485] Lustre: Skipped 2 previous similar messages [16504.204845] Lustre: server umount lustre-MDT0000 complete [16508.313342] LustreError: 523382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1787599482 with bad export cookie 3394565356867415509 [16508.315079] LustreError: MGC192.168.201.103@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16508.344102] LustreError: 523382:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16508.893519] Lustre: server umount lustre-MDT0001 complete [16536.648093] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing set_hostid [16549.157821] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing load_modules_local [16584.531099] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16262 sec ========= 15:25:57 (1787599557) [16586.554618] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 15:25:58 (1787599558) === [16589.986348] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 15:26:02 (1787599562) === [16635.963209] Lustre: DEBUG MARKER: oleg103-server.virtnet: executing unload_modules_local [16638.962687] Key type lgssc unregistered [16639.256696] LNet: 532008:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16639.264148] LNetError: 532008:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16639.282038] LNet: Removed LNI 192.168.201.103@tcp [16640.366932] Key type .llcrypt unregistered [16640.372457] Key type ._llcrypt unregistered