[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 499719920 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2400.000 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524588K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001014] APIC: Switch to symmetric I/O mode setup [ 0.002233] x2apic enabled [ 0.003006] Switched APIC routing to physical x2apic. [ 0.005006] kvm-guest: setup PV IPIs [ 0.007303] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 0.008024] Calibrating delay loop (skipped) preset value.. 4800.00 BogoMIPS (lpj=2400000) [ 0.009028] pid_max: default: 32768 minimum: 301 [ 0.010142] LSM: Security Framework initializing [ 0.011058] Yama: becoming mindful. [ 0.012035] SELinux: Initializing. [ 0.013057] *** VALIDATE selinux *** [ 0.021688] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026475] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028143] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029111] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030099] *** VALIDATE tmpfs *** [ 0.032245] *** VALIDATE proc *** [ 0.033285] *** VALIDATE cgroup *** [ 0.034011] *** VALIDATE cgroup2 *** [ 0.035271] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037152] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038024] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039028] Spectre V2 : User space: Vulnerable [ 0.040009] Speculative Store Bypass: Vulnerable [ 0.042747] debug: unmapping init [mem 0xffffffffa9659000-0xffffffffa9660fff] [ 0.045230] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046700] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047026] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049014] ... generic registers: 4 [ 0.050013] ... value mask: 0000ffffffffffff [ 0.051014] ... max period: 00007fffffffffff [ 0.052012] ... fixed-purpose events: 3 [ 0.052990] ... event mask: 000000070000000f [ 0.053326] rcu: Hierarchical SRCU implementation. [ 0.055435] smp: Bringing up secondary CPUs ... [ 0.056576] x86: Booting SMP configuration: [ 0.057037] .... node #0, CPUs: #1 #2 #3 [ 0.062157] smp: Brought up 1 node, 4 CPUs [ 0.064013] smpboot: Max logical packages: 1 [ 0.064948] smpboot: Total of 4 processors activated (19200.00 BogoMIPS) [ 0.096645] node 0 deferred pages initialised in 31ms [ 0.100494] devtmpfs: initialized [ 0.102279] x86/mm: Memory block size: 128MB [ 0.106152] gcov: version magic: 0x41383552 [ 0.108261] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.109097] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.110290] pinctrl core: initialized pinctrl subsystem [ 0.111206] [ 0.111849] ************************************************************* [ 0.112015] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.113013] ** ** [ 0.114007] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.115010] ** ** [ 0.116012] ** This means that this kernel is built to expose internal ** [ 0.117008] ** IOMMU data structures, which may compromise security on ** [ 0.118009] ** your system. ** [ 0.119010] ** ** [ 0.120011] ** If you see this message and you are not debugging the ** [ 0.121014] ** kernel, report this immediately to your vendor! ** [ 0.122011] ** ** [ 0.123008] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.124010] ************************************************************* [ 0.125787] NET: Registered protocol family 16 [ 0.127394] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.129053] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.131043] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.134031] cpuidle: using governor menu [ 0.135868] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.138554] PCI: Using configuration type 1 for base access [ 0.141129] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.151028] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.152012] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.153126] cryptd: max_cpu_qlen set to 1000 [ 0.155195] ACPI: Added _OSI(Module Device) [ 0.156033] ACPI: Added _OSI(Processor Device) [ 0.157012] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.158009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.161167] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.165413] ACPI: Interpreter enabled [ 0.167048] ACPI: PM: (supports S0 S3 S4 S5) [ 0.168008] ACPI: Using IOAPIC for interrupt routing [ 0.169099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.171404] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.181381] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.183039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.186026] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.189159] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.194171] acpiphp: Slot [2] registered [ 0.195068] acpiphp: Slot [5] registered [ 0.197107] acpiphp: Slot [6] registered [ 0.198116] acpiphp: Slot [7] registered [ 0.199078] acpiphp: Slot [8] registered [ 0.200076] acpiphp: Slot [9] registered [ 0.201093] acpiphp: Slot [10] registered [ 0.202119] acpiphp: Slot [3] registered [ 0.203059] acpiphp: Slot [4] registered [ 0.204075] acpiphp: Slot [11] registered [ 0.205063] acpiphp: Slot [12] registered [ 0.207093] acpiphp: Slot [13] registered [ 0.208092] acpiphp: Slot [14] registered [ 0.210109] acpiphp: Slot [15] registered [ 0.212120] acpiphp: Slot [16] registered [ 0.213109] acpiphp: Slot [17] registered [ 0.215085] acpiphp: Slot [18] registered [ 0.216075] acpiphp: Slot [19] registered [ 0.218100] acpiphp: Slot [20] registered [ 0.219095] acpiphp: Slot [21] registered [ 0.220078] acpiphp: Slot [22] registered [ 0.222072] acpiphp: Slot [23] registered [ 0.223096] acpiphp: Slot [24] registered [ 0.225129] acpiphp: Slot [25] registered [ 0.226105] acpiphp: Slot [26] registered [ 0.228095] acpiphp: Slot [27] registered [ 0.229137] acpiphp: Slot [28] registered [ 0.231127] acpiphp: Slot [29] registered [ 0.233110] acpiphp: Slot [30] registered [ 0.234122] acpiphp: Slot [31] registered [ 0.236072] PCI host bridge to bus 0000:00 [ 0.237017] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.239022] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.241016] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.243023] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.245031] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.247029] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.249172] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.252072] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.255292] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.265781] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.270910] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.273018] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.275021] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.278017] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.280479] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.282858] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.285039] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.288881] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.294015] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.306015] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.312014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.318973] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.326019] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.336016] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.358019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.373156] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.381017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.391018] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.411019] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.420485] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.431019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.436016] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.456018] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.472000] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.477018] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.481014] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.501019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.516798] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.525021] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.531015] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.552013] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.564964] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.572017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.581014] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.596018] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.606569] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.608241] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.610239] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.611184] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.612125] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.615166] iommu: Default domain type: Passthrough [ 0.617258] SCSI subsystem initialized [ 0.618095] ACPI: bus type USB registered [ 0.619000] usbcore: registered new interface driver usbfs [ 0.619000] usbcore: registered new interface driver hub [ 0.620046] usbcore: registered new device driver usb [ 0.621108] pps_core: LinuxPPS API ver. 1 registered [ 0.622006] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.624043] PTP clock support registered [ 0.625130] EDAC MC: Ver: 3.0.0 [ 0.626437] PCI: Using ACPI for IRQ routing [ 0.627576] NetLabel: Initializing [ 0.628006] NetLabel: domain hash size = 128 [ 0.628845] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.630060] NetLabel: unlabeled traffic allowed by default [ 0.632093] vgaarb: loaded [ 0.632742] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.633008] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.641137] clocksource: Switched to clocksource kvm-clock [ 0.735607] VFS: Disk quotas dquot_6.6.0 [ 0.737456] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.739685] *** VALIDATE ramfs *** [ 0.741094] *** VALIDATE hugetlbfs *** [ 0.743729] pnp: PnP ACPI init [ 0.746343] pnp: PnP ACPI: found 6 devices [ 0.767852] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.770835] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.772478] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.774447] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.776383] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.777935] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.779774] NET: Registered protocol family 2 [ 0.781755] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.786084] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.789432] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.795434] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.799175] TCP: Hash tables configured (established 65536 bind 65536) [ 0.801607] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.804677] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.807494] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.810416] NET: Registered protocol family 1 [ 0.812881] RPC: Registered named UNIX socket transport module. [ 0.814921] RPC: Registered udp transport module. [ 0.816125] RPC: Registered tcp transport module. [ 0.817226] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.818919] NET: Registered protocol family 44 [ 0.820168] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.821741] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.823421] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.825491] PCI: CLS 0 bytes, default 64 [ 0.827873] Unpacking initramfs... [ 2.226464] debug: unmapping init [mem 0xffff90423cc54000-0xffff90423ffbffff] [ 2.230723] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.233181] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.236569] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x22983777dd9, max_idle_ns: 440795300422 ns [ 2.799494] Initialise system trusted keyrings [ 2.800898] Key type blacklist registered [ 2.802906] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.814581] zbud: loaded [ 2.819549] *** VALIDATE nfs *** [ 2.820699] *** VALIDATE nfs4 *** [ 2.822205] pstore: using deflate compression [ 2.826633] Platform Keyring initialized [ 2.939242] NET: Registered protocol family 38 [ 2.940870] Key type asymmetric registered [ 2.942187] Asymmetric key parser 'x509' registered [ 2.944107] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.946770] io scheduler mq-deadline registered [ 2.947985] io scheduler kyber registered [ 2.949306] io scheduler bfq registered [ 2.951149] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.954842] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.958274] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.961658] ACPI: Power Button [PWRF] [ 2.969636] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.977115] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.991830] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.999341] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.022346] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.049208] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.079878] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.084105] Non-volatile memory driver v1.3 [ 3.085524] Linux agpgart interface v0.103 [ 3.118985] virtio_blk virtio1: [vda] 149984 512-byte logical blocks (76.8 MB/73.2 MiB) [ 3.121700] vda: detected capacity change from 0 to 76791808 [ 3.136337] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.139409] vdb: detected capacity change from 0 to 1073741824 [ 3.154747] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.157825] vdc: detected capacity change from 0 to 2621440000 [ 3.173832] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.177251] vdd: detected capacity change from 0 to 2621440000 [ 3.193502] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.196859] vde: detected capacity change from 0 to 4294967296 [ 3.213829] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.217176] vdf: detected capacity change from 0 to 4294967296 [ 3.224692] libphy: Fixed MDIO Bus: probed [ 3.253414] usbcore: registered new interface driver usbserial_generic [ 3.256010] usbserial: USB Serial support registered for generic [ 3.258762] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.265675] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.271904] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.283438] mousedev: PS/2 mouse device common for all mice [ 3.292520] rtc_cmos 00:05: RTC can wake from S4 [ 3.327167] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.331094] rtc_cmos 00:05: registered as rtc0 [ 3.335369] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.337637] intel_pstate: CPU model not supported [ 3.361168] hid: raw HID events driver (C) Jiri Kosina [ 3.367162] usbcore: registered new interface driver usbhid [ 3.369432] usbhid: USB HID core driver [ 3.370372] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.371430] drop_monitor: Initializing network drop monitor service [ 3.371549] Initializing XFRM netlink socket [ 3.371948] NET: Registered protocol family 10 [ 3.373350] Segment Routing with IPv6 [ 3.373385] NET: Registered protocol family 17 [ 3.373730] mpls_gso: MPLS GSO support [ 3.387341] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.413818] RAS: Correctable Errors collector initialized. [ 3.416111] AVX version of gcm_enc/dec engaged. [ 3.417887] AES CTR mode by8 optimization enabled [ 3.618773] sched_clock: Marking stable (3618657643, 0)->(4555115923, -936458280) [ 3.625347] registered taskstats version 1 [ 3.629332] Loading compiled-in X.509 certificates [ 3.632838] zswap: loaded using pool lzo/zbud [ 3.701248] Key type big_key registered [ 3.767201] Key type encrypted registered [ 3.770333] ima: No TPM chip found, activating TPM-bypass! [ 3.773789] ima: Allocated hash algorithm: sha1 [ 3.781577] ima: No architecture policies found [ 3.786636] evm: Initialising EVM extended attributes: [ 3.790728] evm: security.selinux [ 3.793239] evm: security.ima [ 3.795239] evm: security.capability [ 3.797269] evm: HMAC attrs: 0x1 [ 3.801543] rtc_cmos 00:05: setting system clock to 2026-09-09 16:06:09 UTC (1788969969) [ 3.813226] debug: unmapping init [mem 0xffffffffaa603000-0xffffffffaa7fffff] [ 3.819255] debug: unmapping init [mem 0xffffffffa9382000-0xffffffffa9658fff] [ 3.831171] Write protecting the kernel read-only data: 28672k [ 3.846634] debug: unmapping init [mem 0xffffffffa7a03000-0xffffffffa7bfffff] [ 3.859137] debug: unmapping init [mem 0xffffffffa8314000-0xffffffffa83fffff] [ 3.910880] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.922455] systemd[1]: Detected virtualization kvm. [ 3.927805] systemd[1]: Detected architecture x86-64. [ 3.930625] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 4.028472] systemd[1]: No hostname configured. [ 4.033731] systemd[1]: Set hostname to . [ 4.039271] random: systemd: uninitialized urandom read (16 bytes read) [ 4.041619] systemd[1]: Initializing machine ID from random generator. [ 4.264518] random: ln: uninitialized urandom read (6 bytes read) [ 4.651337] random: systemd: uninitialized urandom read (16 bytes read) [ 4.654819] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 4.667705] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 4.677585] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Local File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Timers. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Slices. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. Starting Setup Virtual Console... [ OK ] Reached target Paths. Starting Apply Kernel Variables... Starting Create Volatile Files and Directories... Starting Journal Service... [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 6.283621] device-mapper: uevent: version 1.0.3 [ 6.287445] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 8.519549] random: fast init done [ 8.638897] virtio_net virtio0 ens2: renamed from eth0 [ 8.961363] scsi host0: ata_piix [ 9.056607] scsi host1: ata_piix [ 9.058371] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 9.060931] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 13.315989] random: crng init done [ 13.319158] random: 7 urandom warning(s) missed due to ratelimiting [ 15.030595] dracut-initqueue[583]: RTNETLINK answers: File exists [ 16.114928] hrtimer: interrupt took 3857860 ns Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 17.202209] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 20.614792] printk: systemd: 26 output lines suppressed due to ratelimiting [ 21.695703] SELinux: Disabled at runtime. [ 21.850469] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 21.868887] systemd[1]: Detected virtualization kvm. [ 21.870907] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 23.599506] systemd[1]: initrd-switch-root.service: Succeeded. [ 23.608126] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 23.617429] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 23.630836] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 23.640483] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 23.662933] systemd[1]: Starting Journal Service... Starting Journal Service... [ 23.680310] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... Mounting POSIX Message Queue File System... [ OK ] Listening on Process Core Dump Socket. Activating swap /dev/disk/by-label/SWAP... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Reached target Paths. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd File Systems. [ OK ] Stopped target Initrd Root File System. [ 23.884703] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Kernel Debug File System... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Listening on udev Kernel Socket. Mounting Huge Pages File System... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Created slice system-getty.slice. [ OK ] Reached target RPC Port Mapper. [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on initctl Compatibility Named Pipe. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted Kernel Debug File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Apply Kernel Variables. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 25.167677] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 26.315125] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 26.369678] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 26.950937] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 26.984312] EDAC sbridge: Ver: 1.1.2 [ 30.298343] Key type dns_resolver registered [* ] A start job is running for Configur…-only root support (7s / no limit)[ 31.234694] NFS: Registering the id_resolver key type [ 31.241383] Key type id_resolver registered [ 31.250568] Key type id_legacy registered [** ] A start job is running for Configur…-only root support (8s / no limit) [ OK ] Started Configure read-only root support. [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... Starting Load/Save Random Seed... [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Load/Save Random Seed. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Started dnf makecache --timer. [ OK ] Reached target Timers. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started irqbalance daemon. Starting Restore /run/initramfs on shutdown... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Dynamic System Tuning Daemon... [ OK ] Started Login Service. Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Command Scheduler. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting System Logging Service... Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg106-server login: [ 95.873648] libcfs: loading out-of-tree module taints kernel. [ 95.978745] Key type ._llcrypt registered [ 95.981690] Key type .llcrypt registered [ 96.146638] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 117.325371] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 119.363230] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 119.384588] alg: No test for adler32 (adler32-zlib) [ 120.973781] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 122.092622] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 123.920352] Key type lgssc registered [ 125.743115] Lustre: Echo OBD driver; http://www.lustre.org/ [ 148.271943] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 196.912797] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 212.927517] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 212.975633] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 214.530933] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 214.556393] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 214.696730] Lustre: lustre-MDT0000: new disk, initializing [ 214.872131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 214.890074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 220.125537] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 235.723072] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 235.941484] Lustre: 6516:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 235.991706] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 235.996491] Lustre: Skipped 1 previous similar message [ 236.074647] Lustre: lustre-MDT0001: new disk, initializing [ 236.129650] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 236.160764] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 236.174208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 241.669570] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 247.286689] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 259.085423] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 259.566980] Lustre: lustre-OST0000: new disk, initializing [ 259.570111] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 259.584126] Lustre: 8453:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 259.671277] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 267.746492] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 269.409327] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 269.412492] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 269.661928] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 288.619519] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 288.809538] Lustre: lustre-OST0001: new disk, initializing [ 288.816263] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 288.824321] Lustre: 9529:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 288.919308] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 296.536868] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 296.559413] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 296.627062] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 297.745430] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 311.153704] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 318.905775] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 326.479871] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing check_logdir /tmp/testlogs/ [ 332.420308] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing yml_node [ 337.543126] Lustre: DEBUG MARKER: Client: 2.17.58.42 [ 340.349502] Lustre: DEBUG MARKER: MDS: 2.17.58.42 [ 343.157279] Lustre: DEBUG MARKER: OSS: 2.17.58.42 [ 345.284653] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Wed Sep 9 12:11:48 EDT 2026 [ 365.636114] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 367.724721] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 383.970606] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 383.973974] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 383.983390] Lustre: Skipped 1 previous similar message [ 389.090573] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 389.098727] Lustre: Skipped 6 previous similar messages [ 394.215845] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 394.223858] Lustre: Skipped 3 previous similar messages [ 398.304231] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 398.500515] Lustre: server umount lustre-MDT0000 complete [ 402.591573] LustreError: 10344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970368 with bad export cookie 12967296112002420796 [ 402.595403] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 402.604733] LustreError: 10344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 404.451904] LustreError: 6522:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 404.453520] Lustre: lustre-MDT0001-lwp-OST0001: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 404.454930] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 404.469204] LustreError: 6522:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 404.509665] Lustre: Skipped 3 previous similar messages [ 407.457393] LustreError: 6523:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 407.475449] LustreError: 6523:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 409.007296] Lustre: server umount lustre-MDT0001 complete [ 413.185924] Lustre: server umount lustre-OST0000 complete [ 417.559376] Lustre: server umount lustre-OST0001 complete [ 424.981229] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 436.016736] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 471.003665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 471.300370] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 471.354376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 471.501946] Lustre: lustre-MDT0000: new disk, initializing [ 471.618428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 471.666370] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 476.667377] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 488.858212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 489.027914] Lustre: 15083:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 489.149232] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 489.163381] Lustre: Skipped 1 previous similar message [ 489.355678] Lustre: lustre-MDT0001: new disk, initializing [ 489.521634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 489.642703] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 489.661806] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 494.821362] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 506.255373] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 513.000713] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 519.872210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 520.267681] Lustre: lustre-OST0000: new disk, initializing [ 520.271545] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 520.279737] Lustre: 17055:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 520.405641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 521.825854] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 521.838644] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 521.918757] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 527.096144] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 536.772350] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 541.459201] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 541.657628] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 546.271436] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 546.480849] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 552.934024] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 552.935358] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 552.947290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 552.947301] Lustre: Skipped 3 previous similar messages [ 552.953098] Lustre: Skipped 1 previous similar message [ 552.982753] LustreError: Skipped 1 previous similar message [ 555.006503] Lustre: server umount lustre-OST0000 complete [ 561.122132] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 561.126494] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 561.139922] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 561.149819] Lustre: Skipped 1 previous similar message [ 565.810844] Lustre: server umount lustre-MDT0000 complete [ 568.295203] LustreError: 15097:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 568.315530] LustreError: 15097:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 570.048947] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970535 with bad export cookie 12967296112002422315 [ 570.049841] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 570.057389] LustreError: 15078:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 570.506951] Lustre: server umount lustre-MDT0001 complete [ 575.061638] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 12:15:38 (1788970538) [ 577.800465] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 579.753130] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 12:15:43 (1788970543) [ 586.520852] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.192172] LustreError: 19465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 587.260169] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 591.528728] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 592.362546] LustreError: 19466:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.322722] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 601.573752] LustreError: 19465:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 601.586992] LustreError: 19465:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 606.312457] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 614.685680] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 621.498977] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 629.440142] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 636.631976] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 645.503266] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 657.894473] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 657.915155] Lustre: Skipped 1 previous similar message [ 657.921530] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 657.932866] Lustre: Skipped 3 previous similar messages [ 663.120312] LustreError: 22046:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 663.140468] LustreError: 22046:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 663.436305] Lustre: server umount lustre-OST0000 complete [ 668.644672] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 668.654646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 668.666768] Lustre: Skipped 1 previous similar message [ 675.000279] Lustre: server umount lustre-MDT0000 complete [ 679.121342] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788970644 with bad export cookie 12967296112002423197 [ 679.123237] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 679.133880] LustreError: 19445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 679.687813] Lustre: server umount lustre-MDT0001 complete [ 693.881772] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 697.940044] Key type lgssc unregistered [ 698.204884] LNet: 23000:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 698.210195] LNetError: 23000:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 698.229825] LNet: Removed LNI 192.168.201.106@tcp [ 699.360906] Key type .llcrypt unregistered [ 699.362982] Key type ._llcrypt unregistered [ 725.254535] Key type ._llcrypt registered [ 725.256488] Key type .llcrypt registered [ 725.355774] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 726.830873] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 726.882743] alg: No test for adler32 (adler32-zlib) [ 727.981924] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 728.297879] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 729.992210] Key type lgssc registered [ 731.143617] Lustre: Echo OBD driver; http://www.lustre.org/ [ 739.495355] loop: module loaded [ 739.608559] loop0: detected capacity change from 0 to 204800000 [ 739.820481] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 739.826483] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 740.089160] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 747.848537] loop0: detected capacity change from 0 to 204800000 [ 748.122042] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 748.138132] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 748.149128] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 753.955896] loop0: detected capacity change from 0 to 204800000 [ 759.545329] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 759.572175] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 761.024793] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 761.083049] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 761.205769] Lustre: lustre-MDT0000: new disk, initializing [ 761.306682] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 761.332474] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 766.712467] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 776.459820] loop1: detected capacity change from 0 to 204800000 [ 782.176378] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 782.562837] Lustre: lustre-OST07c6: new disk, initializing [ 782.577653] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 782.585586] Lustre: Skipped 1 previous similar message [ 782.595895] Lustre: 26186:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 782.737874] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 791.090097] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 792.700976] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 792.706771] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 792.739754] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 797.173039] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 797.178544] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 797.182049] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 797.185383] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 797.189036] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 797.192175] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 809.154544] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 809.172953] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 809.185852] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 809.196620] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 809.208037] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 809.222207] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 814.851477] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 814.859759] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 814.867737] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 814.871309] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.877362] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 814.891506] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.895827] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 814.906848] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.914173] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 814.918265] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 814.922629] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 814.926548] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.057220] Lustre: 26715:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 818.064577] Lustre: 26715:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 818.068302] Lustre: 26715:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 818.072760] Lustre: 26715:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.076773] Lustre: 26715:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 818.080295] Lustre: 26715:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.083852] Lustre: 26715:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 818.087530] Lustre: 26715:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.092065] Lustre: 26715:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 818.096306] Lustre: 26715:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 818.098827] Lustre: 26715:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 818.102720] Lustre: 26715:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 823.324115] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 823.336591] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 823.348919] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 823.360646] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 823.370496] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 823.376036] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 823.388103] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 823.399251] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 823.404851] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 823.411664] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 823.417079] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 823.422853] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 831.594993] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 831.608696] Lustre: 25174:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 831.625122] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 831.633851] Lustre: 25174:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 831.646351] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 831.658458] Lustre: 25174:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 831.666680] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 831.672866] Lustre: 25174:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 831.678830] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 831.683929] Lustre: 25174:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 831.687083] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 831.691344] Lustre: 25174:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 848.184968] Lustre: 25175:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 848.206206] Lustre: 25175:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 7 previous similar messages [ 848.216342] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 848.224579] Lustre: 25175:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 848.235100] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 848.244681] Lustre: 25175:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 848.255133] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 848.266857] Lustre: 25175:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 848.277447] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 848.286293] Lustre: 25175:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 848.297118] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 848.305540] Lustre: 25175:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 7 previous similar messages [ 889.843486] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 889.869140] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 893.550841] Lustre: server umount lustre-OST07c6 complete [ 903.286953] Lustre: server umount lustre-MDT0000 complete [ 921.010027] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 923.917293] Key type lgssc unregistered [ 924.328898] LNet: 28746:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 924.334690] LNetError: 28746:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 924.348829] LNet: Removed LNI 192.168.201.106@tcp [ 925.533786] Key type .llcrypt unregistered [ 925.535953] Key type ._llcrypt unregistered [ 946.621059] Key type ._llcrypt registered [ 946.625588] Key type .llcrypt registered [ 946.734828] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 947.890602] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 948.094521] alg: No test for adler32 (adler32-zlib) [ 949.251976] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 949.456797] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 951.144220] Key type lgssc registered [ 952.416490] Lustre: Echo OBD driver; http://www.lustre.org/ [ 960.394716] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 960.421823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 961.965917] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 966.352642] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 972.261502] LustreError: 30014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 972.285663] LustreError: 30014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 975.904545] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 976.586709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 982.885210] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 992.925670] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 999.171327] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1007.301319] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1007.889108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1016.276876] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1024.465422] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1031.490547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 1052.136194] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1052.150045] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1052.160684] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1052.166163] Lustre: Skipped 1 previous similar message [ 1052.177295] Lustre: Skipped 1 previous similar message [ 1053.774996] Lustre: server umount lustre-OST0000 complete [ 1058.275459] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1058.287764] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1058.304091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1062.372669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1062.373483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1064.585610] Lustre: server umount lustre-MDT0000 complete [ 1067.506532] LustreError: 30014:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1067.544814] LustreError: 30014:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1068.960761] LustreError: 29995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971034 with bad export cookie 4666731975540049803 [ 1068.971097] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1068.977964] LustreError: 29995:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1069.378124] Lustre: server umount lustre-MDT0001 complete [ 1081.686868] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1086.086874] Key type lgssc unregistered [ 1086.529642] LNet: 33770:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1086.536155] LNetError: 33770:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1087.596321] LNet: Removed LNI 192.168.201.106@tcp [ 1089.041510] Key type .llcrypt unregistered [ 1089.043433] Key type ._llcrypt unregistered [ 1111.133228] Key type ._llcrypt registered [ 1111.137735] Key type .llcrypt registered [ 1111.254379] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1112.761673] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1112.784205] alg: No test for adler32 (adler32-zlib) [ 1113.912990] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1114.210248] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1115.912237] Key type lgssc registered [ 1117.195257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1126.081091] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1126.111846] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1127.918820] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1132.080265] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1138.153377] LustreError: 35036:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1138.175652] LustreError: 35036:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1141.415570] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1141.880459] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1146.576794] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1154.868775] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1161.272886] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1168.808477] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1169.106972] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1173.251757] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1175.807845] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1182.347640] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1197.027857] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1197.034370] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1197.042505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1197.050611] Lustre: Skipped 1 previous similar message [ 1198.537976] Lustre: server umount lustre-OST0000 complete [ 1207.269829] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1207.277086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1207.289391] Lustre: Skipped 1 previous similar message [ 1207.322281] Lustre: Skipped 2 previous similar messages [ 1210.062755] Lustre: server umount lustre-MDT0000 complete [ 1212.386429] LustreError: 35037:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1212.411981] LustreError: 35037:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1215.200599] LustreError: 35017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971180 with bad export cookie 8524738381463071565 [ 1215.207026] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1215.212917] LustreError: 35017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1215.716932] Lustre: server umount lustre-MDT0001 complete [ 1228.896429] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1232.023573] Key type lgssc unregistered [ 1232.324794] LNet: 38592:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1232.332498] LNetError: 38592:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1232.370615] LNet: Removed LNI 192.168.201.106@tcp [ 1233.410353] Key type .llcrypt unregistered [ 1233.413112] Key type ._llcrypt unregistered [ 1262.230351] Key type ._llcrypt registered [ 1262.237197] Key type .llcrypt registered [ 1262.322261] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1263.671309] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1263.812321] alg: No test for adler32 (adler32-zlib) [ 1264.818338] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1265.040993] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1266.704362] Key type lgssc registered [ 1267.763590] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1275.882745] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1275.903847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1277.495459] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1282.516342] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1287.663340] LustreError: 40010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1287.677368] LustreError: 40010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1292.771402] LustreError: 40009:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1293.473231] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1294.131207] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1299.166821] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1308.876221] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1315.793931] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1323.661442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1324.180155] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1332.556398] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1341.004716] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1347.706035] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1357.345443] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 1357.985191] Lustre: server umount lustre-MDT0000 complete [ 1358.313447] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1358.315524] LustreError: 40010:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.341943] Lustre: Skipped 2 previous similar messages [ 1358.363796] LustreError: 40010:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1361.950657] LustreError: 39989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971327 with bad export cookie 15271305642018695052 [ 1361.951927] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1361.963088] LustreError: 39989:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1362.428563] LustreError: 42075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1362.463138] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [ 1363.443281] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1363.471286] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1364.521329] Lustre: server umount lustre-MDT0001 complete [ 1376.720231] LustreError: 42614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.6@tcp arrived at 1788971342 with bad export cookie 15271305642018695717 [ 1376.729119] LustreError: 42614:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1385.252499] Lustre: server umount lustre-OST0000 complete [ 1404.481618] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1407.425663] Key type lgssc unregistered [ 1407.733751] LNet: 43895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1407.739421] LNetError: 43895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1408.807955] LNet: Removed LNI 192.168.201.106@tcp [ 1409.894635] Key type .llcrypt unregistered [ 1409.898903] Key type ._llcrypt unregistered [ 1437.946486] Key type ._llcrypt registered [ 1437.949376] Key type .llcrypt registered [ 1438.082126] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1439.187682] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1439.288221] alg: No test for adler32 (adler32-zlib) [ 1440.574445] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1440.776708] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1442.416343] Key type lgssc registered [ 1443.580813] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1451.694824] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1451.722614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1453.208315] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1457.653419] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1462.765127] LustreError: 45307:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1462.783204] LustreError: 45307:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1466.956258] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1467.269890] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1471.908658] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1480.605831] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1487.117320] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1495.564612] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1495.989894] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1498.041595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1502.990431] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1511.086585] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1518.564068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1518.572626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1518.591709] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1519.125259] Lustre: lustre-OST0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 1521.645793] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1521.650807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1524.222740] Lustre: lustre-OST0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 1524.239550] Lustre: Skipped 1 previous similar message [ 1524.571666] Lustre: server umount lustre-OST0000 complete [ 1544.162259] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1544.177745] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1544.185583] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1547.235117] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1549.343625] Lustre: server umount lustre-MDT0000 complete [ 1552.356510] LustreError: 45302:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1552.392633] LustreError: 45302:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 1554.074167] LustreError: 45288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971519 with bad export cookie 14996134663067645767 [ 1554.076111] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1554.088931] LustreError: 45288:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1560.839474] Lustre: server umount lustre-MDT0001 complete [ 1577.200403] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1580.318073] Key type lgssc unregistered [ 1580.616423] LNet: 49016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1580.623496] LNetError: 49016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1580.636790] LNet: Removed LNI 192.168.201.106@tcp [ 1581.571232] Key type .llcrypt unregistered [ 1581.576736] Key type ._llcrypt unregistered [ 1607.307273] Key type ._llcrypt registered [ 1607.309634] Key type .llcrypt registered [ 1607.422240] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1608.857780] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1608.891178] alg: No test for adler32 (adler32-zlib) [ 1609.884210] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1610.115792] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1611.824291] Key type lgssc registered [ 1612.959854] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1620.132454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1620.150762] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1621.718614] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1626.593894] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1632.234730] LustreError: 50432:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1632.265302] LustreError: 50432:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1635.637938] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1636.157519] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1640.703263] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1649.177409] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1655.672669] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1664.294648] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1664.895975] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1666.931712] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1671.718622] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1681.321669] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1687.587683] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1691.660555] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1704.298326] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0912h34m27s [ 1706.250253] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1709.577478] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 1710.651387] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 1712.101408] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1712.105082] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1712.139461] Lustre: Skipped 1 previous similar message [ 1714.989503] Lustre: server umount lustre-MDT0000 complete [ 1715.700766] LustreError: 51665:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1718.772754] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1719.276112] LustreError: 50431:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1719.298343] LustreError: 50431:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 1721.845691] LustreError: 50427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1726.829898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1727.052866] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1727.448483] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1727.533100] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1728.499953] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1731.701799] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1736.384791] Lustre: DEBUG MARKER: Wait for df (36708) ... [ 1739.437377] Lustre: DEBUG MARKER: done [ 1747.937377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1747.940733] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1747.958447] LustreError: Skipped 1 previous similar message [ 1747.959142] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1747.959146] Lustre: Skipped 5 previous similar messages [ 1748.026536] Lustre: Skipped 2 previous similar messages [ 1751.713137] Lustre: server umount lustre-OST0000 complete [ 1759.201574] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1759.201594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1759.216135] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1759.226851] Lustre: Skipped 4 previous similar messages [ 1762.436263] Lustre: server umount lustre-MDT0000 complete [ 1766.374870] LustreError: 50428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1766.398120] LustreError: 50428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 1766.725796] LustreError: 50413:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971732 with bad export cookie 13836197365485889748 [ 1766.737319] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1767.084884] Lustre: server umount lustre-MDT0001 complete [ 1780.073646] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 1783.255455] Key type lgssc unregistered [ 1783.539903] LNet: 55356:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1783.545143] LNetError: 55356:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1783.562291] LNet: Removed LNI 192.168.201.106@tcp [ 1784.266054] Key type .llcrypt unregistered [ 1784.275511] Key type ._llcrypt unregistered [ 1829.725201] Key type ._llcrypt registered [ 1829.728221] Key type .llcrypt registered [ 1829.843270] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 1831.414856] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1831.513911] alg: No test for adler32 (adler32-zlib) [ 1832.587767] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 1832.795732] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 1834.448911] Key type lgssc registered [ 1835.497709] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1843.550590] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1843.567465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1844.835483] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1844.852128] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1844.991864] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1849.841717] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1859.699195] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1859.779667] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1859.808840] Lustre: 57566:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1860.033376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1865.129925] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1874.639325] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1881.365348] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1888.874602] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1889.146309] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1889.364801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1895.712501] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1895.972584] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 1904.583132] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1910.872646] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0912h37m54s [ 1912.656577] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1920.328149] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1922.990614] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1923.128545] LustreError: 60008:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1943.602230] Lustre: lustre-MDT0000: Client 71861d3a-df2b-45e5-8ff2-0f304c2ada3b (at 192.168.201.6@tcp) reconnecting [ 1963.216234] LustreError: 60008:0:(mdt_handler.c:2344:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1965.427290] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1967.974768] Lustre: DEBUG MARKER: done [ 1974.716992] Lustre: server umount lustre-OST0000 complete [ 1982.957985] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1982.966881] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1982.968816] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1982.970292] Lustre: Skipped 1 previous similar message [ 1984.044744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1984.049248] Lustre: Skipped 2 previous similar messages [ 1990.117066] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1990.130199] Lustre: Skipped 1 previous similar message [ 1990.330487] Lustre: server umount lustre-MDT0000 complete [ 1994.208350] Lustre: 56887:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788971943/real 1788971943] req@00000000307dbf41 x1875872892114560/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788971959 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1994.228374] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1994.417535] LustreError: 57558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788971960 with bad export cookie 4461673957671744020 [ 1994.421307] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1994.433336] LustreError: 57558:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1994.825873] Lustre: server umount lustre-MDT0001 complete [ 2006.369588] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2009.292774] Key type lgssc unregistered [ 2009.624711] LNet: 61895:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2009.630626] LNetError: 61895:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2009.659325] LNet: Removed LNI 192.168.201.106@tcp [ 2010.397902] Key type .llcrypt unregistered [ 2010.400579] Key type ._llcrypt unregistered [ 2061.051966] Key type ._llcrypt registered [ 2061.053451] Key type .llcrypt registered [ 2061.169849] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2062.701242] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2062.923033] alg: No test for adler32 (adler32-zlib) [ 2064.162896] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2064.555697] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2066.262637] Key type lgssc registered [ 2067.406560] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2071.635502] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2071.662697] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2073.001460] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 2073.034321] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2073.347983] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2074.570022] Lustre: Failing over lustre-MDT0000 [ 2075.116237] Lustre: server umount lustre-MDT0000 complete [ 2084.871794] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 12:40:48 (1788972048) [ 2092.412885] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2092.895050] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.109667] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2109.715342] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2109.816164] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 2109.831342] Lustre: 64690:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2110.051848] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2114.615789] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2122.204543] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2128.341626] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2135.193365] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2135.352452] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2135.613079] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2142.322226] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2143.392604] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2150.918313] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2156.832422] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2163.397802] Lustre: server umount lustre-MDT0000 complete [ 2164.196382] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2164.230815] Lustre: Skipped 1 previous similar message [ 2168.194788] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972133 with bad export cookie 707157182150624412 [ 2168.201840] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2168.210354] LustreError: 64682:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2168.564222] Lustre: server umount lustre-MDT0001 complete [ 2173.034531] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2178.192699] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2185.697827] Lustre: 63612:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788972135/real 1788972135] req@000000000da34d34 x1875873135368960/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788972151 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2185.722495] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2185.734961] Lustre: Skipped 1 previous similar message [ 2187.194709] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2189.728308] Lustre: 63614:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788972139/real 1788972139] req@000000003b4ce412 x1875873135369088/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788972155 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2190.752162] Lustre: 63613:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788972140/real 1788972140] req@00000000de177e87 x1875873135369216/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788972156 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2192.931569] LustreError: 63610:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000de177e87 x1875873135370112/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2193.330686] LustreError: 66542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2193.352016] LustreError: 66542:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2193.439468] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2193.531504] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2194.401046] Lustre: 63612:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788972143/real 1788972143] req@000000009df43a91 x1875873135369472/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788972159 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2195.499110] LustreError: 68464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2195.535715] LustreError: 68464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2198.565586] Lustre: 63612:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788972148/real 1788972148] req@00000000263d48c7 x1875873135369600/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788972164 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2198.618827] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2200.549301] LustreError: 68462:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2200.577245] LustreError: 68462:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2204.655144] LustreError: 68464:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2204.657775] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2204.676259] LustreError: 68464:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2207.447176] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2207.744653] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2212.857581] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2213.145975] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2222.884711] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2230.917538] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2237.712480] Lustre: server umount lustre-MDT0000 complete [ 2238.433268] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2238.438731] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2238.443086] LustreError: 68836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2238.447048] Lustre: Skipped 2 previous similar messages [ 2238.486619] LustreError: 68836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2241.131948] LustreError: 65428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972206 with bad export cookie 707157182150626687 [ 2241.132607] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2241.145964] LustreError: 65428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2243.626379] Lustre: server umount lustre-MDT0001 complete [ 2249.884825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2259.244463] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2259.936195] Lustre: 63612:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788972209/real 1788972209] req@00000000132490be x1875873135396096/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788972225 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2259.965419] Lustre: 63612:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2259.973764] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2266.147620] LustreError: 63610:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001151bf69 x1875873135397248/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2266.418600] LustreError: 66542:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2266.478496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2266.528856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2270.757328] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2278.885097] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2278.946370] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2283.506392] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2284.539942] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2290.854596] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2297.049429] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2303.994295] Lustre: server umount lustre-MDT0000 complete [ 2304.998517] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2305.011355] LustreError: 71144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2305.016240] Lustre: Skipped 2 previous similar messages [ 2305.043570] LustreError: 71144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 8 previous similar messages [ 2308.178707] LustreError: 64681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972273 with bad export cookie 707157182150628066 [ 2308.179763] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2308.188936] LustreError: 64681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2308.637696] Lustre: server umount lustre-MDT0001 complete [ 2315.685364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2320.088680] Lustre: DEBUG MARKER: files compared the same [ 2331.889453] Lustre: server umount lustre-OST0000 complete [ 2352.706884] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2356.529534] Key type lgssc unregistered [ 2356.902584] LNet: 74419:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2356.913689] LNetError: 74419:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2356.934523] LNet: Removed LNI 192.168.201.106@tcp [ 2358.051641] Key type .llcrypt unregistered [ 2358.053485] Key type ._llcrypt unregistered [ 2385.459827] Key type ._llcrypt registered [ 2385.462167] Key type .llcrypt registered [ 2385.734147] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2388.156669] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2388.242721] alg: No test for adler32 (adler32-zlib) [ 2389.593575] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2390.058978] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2391.811771] Key type lgssc registered [ 2394.027309] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2405.677264] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2405.705726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2407.126877] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2411.269669] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug +malloc all [ 2416.618581] LustreError: 75834:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2416.647976] LustreError: 75834:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2421.107453] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2421.489886] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2426.242860] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug +malloc all [ 2433.751454] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2440.834304] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2448.153119] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2448.501219] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2451.568178] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2455.386866] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug +malloc all [ 2464.022722] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2477.540297] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2477.541733] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2477.542050] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2477.575143] Lustre: Skipped 1 previous similar message [ 2482.665107] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2482.690394] Lustre: Skipped 1 previous similar message [ 2483.393458] Lustre: server umount lustre-OST0000 complete [ 2487.777284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2487.781190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2487.808133] Lustre: Skipped 1 previous similar message [ 2487.813885] Lustre: Skipped 1 previous similar message [ 2492.899726] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2492.910964] Lustre: Skipped 2 previous similar messages [ 2494.364124] Lustre: server umount lustre-MDT0000 complete [ 2498.029456] LustreError: 75831:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2498.055476] LustreError: 75831:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2499.121431] LustreError: 78436:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972464 with bad export cookie 14123712303927412732 [ 2499.132794] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2499.599628] Lustre: server umount lustre-MDT0001 complete [ 2514.526994] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2517.301463] Key type lgssc unregistered [ 2517.675602] LNet: 79389:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2517.686756] LNetError: 79389:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2517.708871] LNet: Removed LNI 192.168.201.106@tcp [ 2518.812640] Key type .llcrypt unregistered [ 2518.817414] Key type ._llcrypt unregistered [ 2548.418097] Key type ._llcrypt registered [ 2548.420239] Key type .llcrypt registered [ 2548.531848] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2549.582381] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2549.662536] alg: No test for adler32 (adler32-zlib) [ 2550.618681] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2550.796279] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2552.440273] Key type lgssc registered [ 2553.253806] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2562.768736] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2562.799994] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2578.528408] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2583.649417] LustreError: 80781:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.106@tcp: failed processing log, type 4: rc = -110 [ 2610.400281] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2616.933245] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2625.034302] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2634.235599] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.398606] Lustre: *** cfs_fail_loc=706, val=0*** [ 2635.069338] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2635.166485] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2640.113851] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2640.356529] LustreError: 80806:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2640.389088] LustreError: 80806:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2644.463090] LustreError: 80807:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2644.490830] LustreError: 80807:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2649.582360] LustreError: 80806:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.603092] LustreError: 80806:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2652.447138] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2653.138821] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2657.932173] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2665.733926] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2671.957489] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2676.194592] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2676.202639] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2676.211770] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2678.758153] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2678.759859] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2678.762866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2680.590123] Lustre: server umount lustre-OST0000 complete [ 2688.998993] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2688.999712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.045700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2689.048885] Lustre: Skipped 2 previous similar messages [ 2690.900907] Lustre: server umount lustre-MDT0000 complete [ 2694.118262] LustreError: 81866:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2694.147898] LustreError: 81866:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2695.196789] LustreError: 80787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788972660 with bad export cookie 6666696499713445976 [ 2695.212874] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2695.218238] LustreError: 80787:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2695.779980] Lustre: server umount lustre-MDT0001 complete [ 2715.353948] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 2719.428671] Key type lgssc unregistered [ 2720.033730] LNet: 84414:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2720.054522] LNetError: 84414:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2721.130509] LNet: Removed LNI 192.168.201.106@tcp [ 2722.243404] Key type .llcrypt unregistered [ 2722.246638] Key type ._llcrypt unregistered [ 2750.123088] Key type ._llcrypt registered [ 2750.127062] Key type .llcrypt registered [ 2750.343481] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2752.157546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2752.188252] alg: No test for adler32 (adler32-zlib) [ 2753.305617] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 2753.470884] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 2755.224244] Key type lgssc registered [ 2757.253208] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2767.230500] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2767.253044] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2773.491781] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2783.273392] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2783.930054] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2788.627968] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2789.350170] LustreError: 86516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2789.368000] LustreError: 86516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2793.446914] LustreError: 86519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2793.473198] LustreError: 86519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2798.448903] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2798.568480] LustreError: 86520:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2798.609403] LustreError: 86520:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 2799.030641] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2804.203481] LustreError: 86516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2804.227500] LustreError: 86516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 2808.270514] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2808.360631] LustreError: 86519:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2808.383688] LustreError: 86519:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 2818.206936] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2818.236992] Lustre: MGS: Client e1350c2d-8cc4-4118-a1b0-ef4219353986 (at 0@lo) reconnecting [ 2818.268603] LustreError: 86530:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2818.465428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2818.529736] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2824.155938] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2844.131275] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2844.153365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2844.163097] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2844.190163] Lustre: Skipped 1 previous similar message [ 2847.635878] Lustre: server umount lustre-OST0000 complete [ 2854.372228] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2854.380433] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2854.398989] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2854.424630] Lustre: Skipped 2 previous similar messages [ 2857.828603] Lustre: server umount lustre-MDT0000 complete [ 2859.493054] LustreError: 86516:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2859.516092] LustreError: 86516:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [ 2862.860684] Lustre: server umount lustre-MDT0001 complete [ 2867.413877] Lustre: server umount MGS complete [ 2880.718684] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 12:54:04 (1788972844) [ 2916.488094] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 2928.275414] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 2970.452711] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.639388] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 2983.953582] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2984.370729] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2984.491890] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2984.601205] Lustre: lustre-MDT0001: new disk, initializing [ 2984.696191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2984.730776] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2984.747664] Lustre: 93707:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2984.765315] LustreError: 93707:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2989.838622] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3000.705959] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3000.957954] Lustre: lustre-OST0000: new disk, initializing [ 3000.965215] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3000.976082] Lustre: 94473:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3001.056731] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3007.928973] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3010.209564] Lustre: 94879:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000c556fd98 x1875873857840768/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788972991 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3019.608643] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3027.789828] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3027.804541] Lustre: MGS: Client 315024d6-bea6-458c-a15f-532de1b13164 (at 0@lo) reconnecting [ 3027.835155] LustreError: 93693:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 3027.861401] Lustre: 95519:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3027.891429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3027.956945] Lustre: lustre-MDT0000: new disk, initializing [ 3028.164839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3028.203314] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3028.232657] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3029.282203] Lustre: 95543:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000028c6346b x1875873857847680/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788973010 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 3033.733684] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3034.465912] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 3037.439147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3037.456487] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3038.560607] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3064.291107] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3064.305904] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3064.332762] Lustre: Skipped 1 previous similar message [ 3064.342882] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3069.926373] Lustre: server umount lustre-OST0000 complete [ 3076.066452] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3076.070361] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3076.084848] Lustre: Skipped 1 previous similar message [ 3076.086072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3076.086077] Lustre: Skipped 2 previous similar messages [ 3081.187487] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3081.195952] Lustre: Skipped 2 previous similar messages [ 3081.503100] Lustre: server umount lustre-MDT0000 complete [ 3086.306402] LustreError: 93682:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3086.344305] LustreError: 93682:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3094.669619] Lustre: server umount lustre-MDT0001 complete [ 3099.176670] Lustre: server umount MGS complete [ 3113.561526] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 12:57:56 (1788973076) [ 3135.656289] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 3138.563667] Key type lgssc unregistered [ 3139.014676] LNet: 98043:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3139.033943] LNetError: 98043:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3139.065511] LNet: Removed LNI 192.168.201.106@tcp [ 3140.186561] Key type .llcrypt unregistered [ 3140.191355] Key type ._llcrypt unregistered [ 3163.351217] Key type ._llcrypt registered [ 3163.358863] Key type .llcrypt registered [ 3163.640951] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3165.691333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3165.867196] alg: No test for adler32 (adler32-zlib) [ 3167.524471] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 3167.787307] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 3169.504483] Key type lgssc registered [ 3170.996370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3181.579619] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3181.614562] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3183.015057] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3188.094882] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3193.314644] LustreError: 99274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3193.321912] LustreError: 99274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3198.434470] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3198.438032] LustreError: 99273:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3198.926035] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3204.400110] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3213.261436] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3221.228430] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3230.911163] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3231.452500] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3240.623422] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3250.176435] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3254.010986] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3269.618088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3269.620578] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3269.649576] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3269.686776] Lustre: Skipped 1 previous similar message [ 3272.536758] Lustre: server umount lustre-OST0000 complete [ 3279.842859] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3279.868388] Lustre: Skipped 1 previous similar message [ 3279.882361] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3279.898382] Lustre: Skipped 1 previous similar message [ 3284.392209] Lustre: server umount lustre-MDT0000 complete [ 3284.966662] LustreError: 99274:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3284.990186] LustreError: 99274:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3290.108639] LustreError: 100711:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3290.146588] LustreError: 100711:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3292.625114] LustreError: 99255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973258 with bad export cookie 8567058706213976239 [ 3292.631381] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3292.643416] LustreError: 99255:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3293.612089] Lustre: server umount lustre-MDT0001 complete [ 3309.362729] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 3313.358900] Key type lgssc unregistered [ 3313.752612] LNet: 102844:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3313.766418] LNetError: 102844:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3314.792712] LNet: Removed LNI 192.168.201.106@tcp [ 3315.852622] Key type .llcrypt unregistered [ 3315.856113] Key type ._llcrypt unregistered [ 3337.597531] Key type ._llcrypt registered [ 3337.599761] Key type .llcrypt registered [ 3337.706788] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3339.015838] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3339.057148] alg: No test for adler32 (adler32-zlib) [ 3340.185303] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 3340.456440] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 3342.136222] Key type lgssc registered [ 3343.405229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3350.286900] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3350.297150] LustreError: 103914:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3350.377283] LustreError: 103925:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3350.397451] LustreError: 103914:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4915 [ 3350.413481] LustreError: 103914:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3350.417291] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3350.427300] LustreError: 103914:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3352.053297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3360.849212] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3361.172502] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3366.378384] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3372.450767] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing lsmod [ 3375.591417] LustreError: 105128:0:(tgt_mount.c:2569:server_fill_super()) cfs_race id 716 sleeping [ 3375.603655] LustreError: 105129:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 waking [ 3375.608790] LustreError: 105128:0:(tgt_mount.c:2569:server_fill_super()) cfs_fail_race id 716 awake: rc=4988 [ 3375.618728] LustreError: 105128:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3375.632809] LustreError: 105128:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3375.643678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3376.073676] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3377.132116] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:65) [ 3381.730040] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3381.748481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3381.780323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.266627] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3382.292645] Lustre: Skipped 1 previous similar message [ 3382.313669] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3382.316603] Lustre: Skipped 2 previous similar messages [ 3385.380460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3385.388570] Lustre: Skipped 2 previous similar messages [ 3386.166435] Lustre: server umount lustre-MDT0000 complete [ 3390.412120] LustreError: 103944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973356 with bad export cookie 17278592098770213499 [ 3390.415227] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3390.423878] LustreError: 103944:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3390.464865] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3390.471679] LustreError: 103956:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3390.487707] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 3390.528917] LustreError: 103956:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3391.587230] LustreError: 104392:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007024a13a x1875874473335296/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3391.808459] Lustre: server umount lustre-MDT0001 complete [ 3396.609077] Lustre: server umount lustre-OST0000 complete [ 3408.213678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3409.248726] LustreError: 106195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3409.422375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3414.511263] LustreError: 106196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3416.381611] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3419.620502] LustreError: 106195:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3424.754953] LustreError: 106196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3427.972112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3428.924215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3433.869348] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3443.099870] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3449.375504] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3456.706996] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3457.141267] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3464.015406] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3472.805480] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3479.655663] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:97) [ 3494.369153] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3494.380356] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3494.398693] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3495.394115] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3497.226524] Lustre: server umount lustre-OST0000 complete [ 3505.635477] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3505.641308] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3505.651595] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3505.651606] Lustre: Skipped 2 previous similar messages [ 3505.730964] Lustre: Skipped 2 previous similar messages [ 3515.360141] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3515.719679] Lustre: server umount lustre-MDT0000 complete [ 3515.889860] LustreError: 107184:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3515.927740] LustreError: 107184:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3520.067932] LustreError: 106175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973485 with bad export cookie 17278592098770214430 [ 3520.069969] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3520.080300] LustreError: 106175:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3520.901976] Lustre: server umount lustre-MDT0001 complete [ 3537.356425] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 3541.086669] Key type lgssc unregistered [ 3541.555847] LNet: 109754:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3541.566840] LNetError: 109754:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3541.597739] LNet: Removed LNI 192.168.201.106@tcp [ 3542.825068] Key type .llcrypt unregistered [ 3542.827265] Key type ._llcrypt unregistered [ 3570.412877] Key type ._llcrypt registered [ 3570.416776] Key type .llcrypt registered [ 3570.545902] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3571.968945] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3572.041281] alg: No test for adler32 (adler32-zlib) [ 3573.106321] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 3573.375309] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 3575.112231] Key type lgssc registered [ 3576.192122] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3585.226688] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3585.245953] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3586.649835] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3591.979869] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3596.780265] LustreError: 111170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3596.824353] LustreError: 111170:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3601.893875] LustreError: 111171:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3604.472326] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3605.065193] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3612.481851] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3624.987916] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3633.031123] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3640.864166] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3641.278879] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3647.815657] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3656.688798] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3657.375298] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3664.560659] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3670.288183] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3675.106132] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3675.117434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3675.140994] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.157804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3677.159340] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3677.165174] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3681.177694] Lustre: server umount lustre-OST0000 complete [ 3686.882124] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3686.889815] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3686.908670] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3686.916290] Lustre: Skipped 1 previous similar message [ 3692.120931] Lustre: server umount lustre-MDT0000 complete [ 3692.517037] LustreError: 113011:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3692.543985] LustreError: 113011:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 3696.283107] LustreError: 111150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973661 with bad export cookie 17238400940126262774 [ 3696.289659] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3696.306305] LustreError: 111150:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3696.975707] Lustre: server umount lustre-MDT0001 complete [ 3709.549839] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 3712.916514] Key type lgssc unregistered [ 3713.301160] LNet: 114809:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3713.309462] LNetError: 114809:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3713.327866] LNet: Removed LNI 192.168.201.106@tcp [ 3714.559106] Key type .llcrypt unregistered [ 3714.561182] Key type ._llcrypt unregistered [ 3735.076712] Key type ._llcrypt registered [ 3735.078150] Key type .llcrypt registered [ 3735.157827] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3736.077522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3736.164088] alg: No test for adler32 (adler32-zlib) [ 3737.402933] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 3737.865980] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 3739.530765] Key type lgssc registered [ 3740.935703] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3749.609490] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3749.620692] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3751.325418] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3756.235221] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3760.613389] LustreError: 116029:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3760.633450] LustreError: 116029:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3765.733388] LustreError: 116028:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3766.029497] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3766.452660] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3771.264532] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3779.909386] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3786.402977] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3793.216318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3793.800652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3795.861468] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 3801.213277] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3809.952879] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3823.602675] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3823.607586] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3823.645906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3826.657304] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3826.703901] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3826.720129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3828.705403] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3828.713426] Lustre: Skipped 1 previous similar message [ 3829.362444] Lustre: server umount lustre-OST0000 complete [ 3833.827541] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3833.845585] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3838.433575] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3838.441355] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3838.449953] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3839.680647] Lustre: server umount lustre-MDT0000 complete [ 3843.821958] LustreError: 116010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788973809 with bad export cookie 5739588387323283829 [ 3843.829830] LustreError: 116010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3843.829902] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3844.182807] Lustre: server umount lustre-MDT0001 complete [ 3859.370310] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 3863.897507] Key type lgssc unregistered [ 3864.447581] LNet: 119587:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3864.470049] LNetError: 119587:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3864.507327] LNet: Removed LNI 192.168.201.106@tcp [ 3865.806295] Key type .llcrypt unregistered [ 3865.809673] Key type ._llcrypt unregistered [ 3895.176268] Key type ._llcrypt registered [ 3895.177855] Key type .llcrypt registered [ 3895.251790] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 3896.664546] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3896.703172] alg: No test for adler32 (adler32-zlib) [ 3898.060477] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 3898.504479] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 3900.272974] Key type lgssc registered [ 3901.883946] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3912.244685] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3912.270264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3913.897305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3919.415049] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3923.429258] LustreError: 121006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3923.470389] LustreError: 121006:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 3928.547714] LustreError: 121007:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3929.903108] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3930.278576] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3935.826550] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3944.444497] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3950.541411] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3958.057733] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3958.394020] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3959.438215] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 3965.461214] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 3974.182926] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3981.639787] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3986.066979] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3986.073476] Lustre: Skipped 2 previous similar messages [ 3989.733445] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 3989.796780] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3992.636613] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3992.647353] Lustre: Skipped 2 previous similar messages [ 3998.322478] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3998.334324] Lustre: Skipped 1 previous similar message [ 4005.134610] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.201.6@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 4005.140912] Lustre: Skipped 2 previous similar messages [ 4011.609530] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.201.6@tcp 192.168.0.[2,10]@tcp [ 4017.171358] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.201.6/24@tcp in log lustre-MDT0000 [ 4017.178824] Lustre: Skipped 2 previous similar messages [ 4017.510668] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.6/24@tcp [ 4017.516965] Lustre: Skipped 1 previous similar message [ 4036.580306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4036.582757] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4036.601802] Lustre: Skipped 1 previous similar message [ 4036.624278] Lustre: Skipped 1 previous similar message [ 4038.008503] Lustre: server umount lustre-OST0000 complete [ 4046.823712] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4046.839326] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4046.839551] Lustre: Skipped 1 previous similar message [ 4046.857921] Lustre: Skipped 1 previous similar message [ 4051.938616] LustreError: 121006:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4051.958410] LustreError: 121006:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4052.255069] Lustre: server umount lustre-MDT0000 complete [ 4056.735268] LustreError: 120986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974022 with bad export cookie 2668454711588213061 [ 4056.742082] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4056.751781] LustreError: 120986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4057.133606] Lustre: server umount lustre-MDT0001 complete [ 4069.105554] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4071.627076] Key type lgssc unregistered [ 4071.853573] LNet: 125878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4071.862455] LNetError: 125878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4071.878234] LNet: Removed LNI 192.168.201.106@tcp [ 4072.739986] Key type .llcrypt unregistered [ 4072.741861] Key type ._llcrypt unregistered [ 4101.660729] Key type ._llcrypt registered [ 4101.663733] Key type .llcrypt registered [ 4101.757529] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4103.104130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4103.136096] alg: No test for adler32 (adler32-zlib) [ 4104.146937] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 4104.352523] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 4106.001508] Key type lgssc registered [ 4107.022218] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4114.520378] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4114.533207] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4115.984463] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 4115.990429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4116.005573] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.201.6/24@tcp [ 4120.699191] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4126.179854] LustreError: 127435:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4126.197058] LustreError: 127435:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4129.488925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4129.722467] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 4129.749178] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.201.6/24@tcp [ 4129.795236] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4134.445790] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4142.117682] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4148.977521] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4155.941839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4156.398891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4160.518519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:201 to 0x280000401:225) [ 4162.111686] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4169.575871] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4186.084971] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4186.087407] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4186.088490] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4186.095870] LustreError: Skipped 1 previous similar message [ 4186.129292] Lustre: Skipped 1 previous similar message [ 4190.186955] Lustre: server umount lustre-OST0000 complete [ 4196.321315] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4196.325878] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.336214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4196.336222] Lustre: Skipped 1 previous similar message [ 4196.363208] Lustre: Skipped 1 previous similar message [ 4200.724712] Lustre: server umount lustre-MDT0000 complete [ 4201.443742] LustreError: 129013:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4201.486071] LustreError: 129013:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4204.422451] LustreError: 130271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974170 with bad export cookie 4378891681438001328 [ 4204.424104] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4204.429194] LustreError: 130271:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4204.762428] Lustre: server umount lustre-MDT0001 complete [ 4217.334788] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4220.170825] Key type lgssc unregistered [ 4220.441698] LNet: 131021:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4220.445689] LNetError: 131021:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4220.461515] LNet: Removed LNI 192.168.201.106@tcp [ 4221.280582] Key type .llcrypt unregistered [ 4221.282519] Key type ._llcrypt unregistered [ 4246.906788] Key type ._llcrypt registered [ 4246.914742] Key type .llcrypt registered [ 4247.039833] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 4263.264763] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4264.678676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4264.875095] alg: No test for adler32 (adler32-zlib) [ 4265.951750] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 4266.309649] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 4268.040323] Key type lgssc registered [ 4269.624259] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4310.172107] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4310.225203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4311.591525] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4311.628910] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4311.688923] Lustre: lustre-MDT0000: new disk, initializing [ 4311.784566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4311.807188] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4317.093709] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4328.822733] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4328.978239] Lustre: 134420:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4329.041667] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4329.044856] Lustre: Skipped 1 previous similar message [ 4329.153485] Lustre: lustre-MDT0001: new disk, initializing [ 4329.236249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4329.279172] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4329.296873] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4334.359478] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4344.552184] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4351.238465] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4358.802426] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4359.146227] Lustre: lustre-OST0000: new disk, initializing [ 4359.149969] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4359.156952] Lustre: 136387:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4359.236128] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4365.939765] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4365.950939] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4366.165747] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4366.803678] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4380.229764] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4386.465918] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4386.715094] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4390.382557] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4390.572198] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4397.026706] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4397.039714] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4397.071916] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4397.087812] Lustre: Skipped 1 previous similar message [ 4402.148676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4402.158393] Lustre: Skipped 1 previous similar message [ 4407.264130] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4407.278325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4407.288319] Lustre: Skipped 1 previous similar message [ 4407.409665] Lustre: server umount lustre-OST0000 complete [ 4412.390958] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4412.395069] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4412.422743] Lustre: Skipped 2 previous similar messages [ 4412.453296] Lustre: Skipped 1 previous similar message [ 4417.506240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4417.513745] Lustre: Skipped 1 previous similar message [ 4418.330181] Lustre: server umount lustre-MDT0000 complete [ 4422.133967] LustreError: 134411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974387 with bad export cookie 16695027921073307102 [ 4422.141553] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4422.142981] LustreError: 134411:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4422.443365] Lustre: server umount lustre-MDT0001 complete [ 4430.574481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4431.004155] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4435.000794] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4436.449305] LustreError: 138605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4436.474235] LustreError: 138605:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4440.550483] LustreError: 138606:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4443.940428] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4444.458063] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4449.494408] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4458.626864] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4466.654765] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4476.722177] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4477.177719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4484.081399] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4492.732171] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4498.062632] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4502.648088] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 4507.873734] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4508.246824] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4520.245823] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4520.814025] Lustre: lustre-OST0001: new disk, initializing [ 4520.829131] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4520.857990] Lustre: 142006:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4521.090449] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4529.399756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4529.433149] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4529.611594] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4531.634579] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4543.025518] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4543.379339] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4548.702692] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4548.946245] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4555.048298] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa08b1156f800.ost_server_uuid 50 [ 4556.542931] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa08b1156f800.ost_server_uuid in IDLE state after 0 sec [ 4565.490463] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4565.496290] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4565.499072] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4565.519469] Lustre: Skipped 1 previous similar message [ 4565.556626] LustreError: Skipped 1 previous similar message [ 4579.297438] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4579.533202] Lustre: server umount lustre-OST0001 complete [ 4580.837694] LustreError: 140428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4580.853586] LustreError: 140428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4583.912793] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4583.921815] LustreError: Skipped 1 previous similar message [ 4583.923579] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4583.951930] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4583.956984] Lustre: Skipped 5 previous similar messages [ 4585.967027] LustreError: 140427:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4585.998313] LustreError: 140427:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4591.086854] LustreError: 140428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4591.101023] LustreError: 140428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 4598.240146] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4598.376678] Lustre: server umount lustre-OST0000 complete [ 4603.369856] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4603.377776] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4603.400409] Lustre: Skipped 1 previous similar message [ 4608.851672] Lustre: server umount lustre-MDT0000 complete [ 4611.562702] LustreError: 138605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4611.585639] LustreError: 138605:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4613.674696] LustreError: 138587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974579 with bad export cookie 16695027921073307984 [ 4613.683601] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4613.689495] LustreError: 138587:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4614.299036] Lustre: server umount lustre-MDT0001 complete [ 4635.982141] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4638.874790] Key type lgssc unregistered [ 4639.298398] LNet: 145016:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4639.316265] LNetError: 145016:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4639.332282] LNet: Removed LNI 192.168.201.106@tcp [ 4640.453245] Key type .llcrypt unregistered [ 4640.459313] Key type ._llcrypt unregistered [ 4687.055597] Key type ._llcrypt registered [ 4687.058721] Key type .llcrypt registered [ 4687.207427] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 4706.773233] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 4708.264776] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4708.332386] alg: No test for adler32 (adler32-zlib) [ 4709.360990] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 4709.548071] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 4711.192928] Key type lgssc registered [ 4712.489757] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4753.284743] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4753.328752] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4754.832755] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4754.860749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4754.923284] Lustre: lustre-MDT0000: new disk, initializing [ 4754.998265] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4755.024184] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4760.566594] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4772.618255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4772.796490] Lustre: 149173:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4772.884989] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4772.889730] Lustre: Skipped 1 previous similar message [ 4773.080126] Lustre: lustre-MDT0001: new disk, initializing [ 4773.311105] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4773.418731] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4773.443922] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4778.839823] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4789.872932] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4796.768388] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4803.888390] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4804.115547] Lustre: lustre-OST0000: new disk, initializing [ 4804.118638] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4804.124033] Lustre: 151138:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4804.214680] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4810.336837] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4810.356877] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4810.536931] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4810.886468] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4821.398763] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4827.632618] Lustre: Failing over lustre-OST0000 [ 4827.847147] Lustre: server umount lustre-OST0000 complete [ 4845.037572] Lustre: 147391:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788974794/real 1788974794] req@000000008367ad81 x1875875908847488/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788974810 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4845.083572] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.202764] Lustre: 147392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788974796/real 1788974796] req@0000000031642e61 x1875875908848512/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788974812 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4847.203612] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4847.242807] Lustre: 147392:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4849.196714] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4849.509041] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4849.549722] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4852.578392] Lustre: 147394:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788974802/real 1788974802] req@000000008ffb61e0 x1875875908849792/t0(0) o400->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788974818 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4854.636914] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4856.819144] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4856.819776] Lustre: lustre-OST0000: Recovery over after 0:02, of 3 clients 3 recovered and 0 were evicted. [ 4856.875293] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4857.760469] Lustre: 147393:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788974807/real 1788974807] req@0000000050acd345 x1875875908850816/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788974823 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4857.779864] Lustre: 147393:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 2 previous similar messages [ 4861.143704] Lustre: Failing over lustre-MDT0000 [ 4861.757844] Lustre: server umount lustre-MDT0000 complete [ 4864.994350] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4865.014243] Lustre: Skipped 1 previous similar message [ 4870.115942] LustreError: 149183:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4870.145504] LustreError: 149183:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 7 previous similar messages [ 4870.645410] LustreError: 149179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4875.237530] LustreError: 149178:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4875.266213] LustreError: 149178:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 4880.357080] Lustre: 147392:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788974830/real 1788974830] req@00000000fe6ac34f x1875875908864512/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788974846 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4880.363844] LustreError: 149179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4880.378716] Lustre: 147392:0:(client.c:2504:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 4880.378791] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4880.439608] LustreError: 149179:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4882.221134] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4885.476251] LustreError: 149179:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4885.509819] LustreError: 149179:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 4891.072604] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4891.223675] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4896.257140] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4896.264856] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4896.291898] Lustre: Skipped 1 previous similar message [ 4896.319033] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 4896.402789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4898.597169] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 4911.587626] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4911.595568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4911.612272] Lustre: Skipped 2 previous similar messages [ 4911.637125] Lustre: Skipped 1 previous similar message [ 4916.734380] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4916.745268] Lustre: Skipped 1 previous similar message [ 4921.831737] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4921.854218] Lustre: Skipped 1 previous similar message [ 4923.360150] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4923.747886] Lustre: server umount lustre-OST0000 complete [ 4932.065476] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4932.076357] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4932.077407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4932.097539] Lustre: Skipped 1 previous similar message [ 4935.034786] Lustre: server umount lustre-MDT0000 complete [ 4937.185534] LustreError: 149184:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4937.224105] LustreError: 149184:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [ 4940.118563] LustreError: 149165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788974905 with bad export cookie 2878522956890751631 [ 4940.123024] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4940.129139] LustreError: 149165:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4940.816609] Lustre: server umount lustre-MDT0001 complete [ 4953.607551] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 4956.450418] Key type lgssc unregistered [ 4956.762530] LNet: 154798:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4956.773601] LNetError: 154798:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4956.794448] LNet: Removed LNI 192.168.201.106@tcp [ 4957.638574] Key type .llcrypt unregistered [ 4957.641712] Key type ._llcrypt unregistered [ 5000.977388] Key type ._llcrypt registered [ 5000.979542] Key type .llcrypt registered [ 5001.115974] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5002.385467] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5002.787883] alg: No test for adler32 (adler32-zlib) [ 5003.860520] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 5004.196821] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 5005.880424] Key type lgssc registered [ 5007.095366] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5015.644404] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5015.675239] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5017.251256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5023.514629] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5026.796028] LustreError: 156317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5026.815549] LustreError: 156317:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5031.916420] LustreError: 156318:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5034.995513] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5037.027104] LustreError: 156317:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5039.714671] LustreError: 156346:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 5039.737880] LustreError: 156346:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 5039.755271] LustreError: 156346:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 23, retries 0, failed: rc = -5 [ 5040.278231] Lustre: server umount lustre-MDT0000 complete [ 5056.910168] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 5066.655517] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5106.487385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5106.776995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5106.810987] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5106.888576] Lustre: lustre-MDT0000: new disk, initializing [ 5106.957516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5106.968979] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5112.114347] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5124.698149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5125.028594] Lustre: 159650:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5125.130534] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5125.135917] Lustre: Skipped 1 previous similar message [ 5125.247610] Lustre: lustre-MDT0001: new disk, initializing [ 5125.490529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5125.610449] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5125.627855] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5131.068906] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5142.500940] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5149.351383] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5156.535759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5156.776922] Lustre: lustre-OST0000: new disk, initializing [ 5156.781277] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5156.786225] Lustre: 161618:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5156.866903] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5162.090564] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5162.101037] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5162.234951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5163.202959] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5173.089793] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5178.085766] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 5179.787163] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 13:32:23 (1788975143) [ 5427.682776] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5427.691702] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5427.722447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5428.708483] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5428.709891] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5428.722418] Lustre: Skipped 1 previous similar message [ 5428.745203] Lustre: Skipped 2 previous similar messages [ 5432.877506] Lustre: server umount lustre-MDT0000 complete [ 5433.829060] LustreError: 159662:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5433.853397] LustreError: 159662:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5436.568885] LustreError: 159644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788975402 with bad export cookie 11645317005885469755 [ 5436.574255] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5436.585058] LustreError: 159644:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5438.946215] LustreError: 162832:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5438.948210] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5438.981082] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5441.953882] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5442.992643] Lustre: server umount lustre-MDT0001 complete [ 5453.649327] Lustre: server umount lustre-OST0000 complete [ 5462.352100] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 5473.673974] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5508.186805] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5508.371223] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5508.394391] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5508.477742] Lustre: lustre-MDT0000: new disk, initializing [ 5508.577773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5508.606403] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5512.716425] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5524.589934] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5524.773868] Lustre: 166284:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5524.854324] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5524.858882] Lustre: Skipped 1 previous similar message [ 5525.044934] Lustre: lustre-MDT0001: new disk, initializing [ 5525.167786] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5525.224572] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5525.250111] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5530.875682] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5541.371091] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5548.361184] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5555.606875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5555.858951] Lustre: lustre-OST0000: new disk, initializing [ 5555.861959] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5555.866714] Lustre: 168253:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5555.953491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5561.954378] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5561.966865] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5562.117484] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5563.383090] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5572.891587] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5578.958936] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5579.166540] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5584.299903] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5584.714814] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5587.945334] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5587.946163] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5587.949965] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5587.957303] LustreError: Skipped 1 previous similar message [ 5587.987207] Lustre: Skipped 1 previous similar message [ 5598.180884] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5598.195437] Lustre: Skipped 4 previous similar messages [ 5601.248248] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5601.505271] Lustre: server umount lustre-OST0000 complete [ 5607.395187] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5607.405253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5612.545384] Lustre: server umount lustre-MDT0000 complete [ 5613.568807] LustreError: 166294:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5613.611553] LustreError: 166294:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 5616.985118] LustreError: 166274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788975582 with bad export cookie 11645317005885628592 [ 5616.995864] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5617.015847] LustreError: 166274:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5617.672180] Lustre: server umount lustre-MDT0001 complete [ 5627.398639] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:39:50 (1788975590) [ 5647.426674] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 5661.787876] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5706.328029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5706.716905] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5706.756368] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5706.816591] Lustre: lustre-MDT0000: new disk, initializing [ 5706.889918] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5706.903228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5712.226871] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5724.552713] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5724.666543] Lustre: 173035:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5724.880731] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5728.845555] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5741.626876] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5748.098866] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5755.512768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5755.716183] Lustre: lustre-OST0000: new disk, initializing [ 5755.718946] Lustre: Skipped 1 previous similar message [ 5755.722565] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5755.726282] Lustre: Skipped 2 previous similar messages [ 5755.731314] Lustre: 175011:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5762.139349] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5764.677640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5764.692909] Lustre: Skipped 1 previous similar message [ 5764.701729] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5764.797151] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5771.089745] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5780.449120] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5780.451319] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5780.463378] LustreError: Skipped 1 previous similar message [ 5780.469057] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5780.485134] Lustre: Skipped 2 previous similar messages [ 5780.491483] Lustre: Skipped 3 previous similar messages [ 5783.576366] Lustre: server umount lustre-OST0000 complete [ 5793.969970] Lustre: server umount lustre-MDT0000 complete [ 5795.811346] LustreError: 175684:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5795.829016] LustreError: 175684:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5798.549631] LustreError: 173026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788975764 with bad export cookie 11645317005885629474 [ 5798.551057] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5798.562864] LustreError: 173026:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5798.831569] Lustre: server umount lustre-MDT0001 complete [ 5809.473867] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 13:42:52 (1788975772) [ 5828.357061] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 5839.697805] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 5877.377568] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5877.687794] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5877.711069] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5877.763403] Lustre: lustre-MDT0000: new disk, initializing [ 5877.819415] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5877.823899] Lustre: Skipped 2 previous similar messages [ 5877.834394] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5882.058338] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5892.887871] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5893.098402] Lustre: 179336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5893.416177] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5899.328337] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5911.266901] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5916.896322] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5925.430249] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5925.705098] Lustre: 181313:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5931.562261] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5931.727352] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5932.586860] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 5943.674772] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5952.482362] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5952.493370] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5952.497351] LustreError: Skipped 1 previous similar message [ 5952.502708] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5952.518941] Lustre: Skipped 3 previous similar messages [ 5952.546626] Lustre: Skipped 4 previous similar messages [ 5958.701067] Lustre: server umount lustre-OST0000 complete [ 5965.281858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5969.461249] Lustre: server umount lustre-MDT0000 complete [ 5972.962227] LustreError: 179342:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5972.996860] LustreError: 179342:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 5973.572132] LustreError: 179329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788975939 with bad export cookie 11645317005885630552 [ 5973.581867] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5973.592287] LustreError: 179329:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5974.022875] Lustre: server umount lustre-MDT0001 complete [ 5986.237679] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 5989.453945] Key type lgssc unregistered [ 5989.812352] LNet: 183169:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5989.818617] LNetError: 183169:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5989.849412] LNet: Removed LNI 192.168.201.106@tcp [ 5990.679442] Key type .llcrypt unregistered [ 5990.682181] Key type ._llcrypt unregistered [ 6017.172382] Key type ._llcrypt registered [ 6017.173870] Key type .llcrypt registered [ 6017.259762] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6018.453676] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6018.515183] alg: No test for adler32 (adler32-zlib) [ 6019.634724] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 6019.927388] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 6021.600267] Key type lgssc registered [ 6022.892603] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6030.524636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6030.575185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6032.201457] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6037.495705] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6041.580349] LustreError: 184584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6041.608582] LustreError: 184584:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6046.709165] LustreError: 184583:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6046.919110] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6047.501204] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6052.368849] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6060.891312] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6067.860847] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6075.626932] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6076.042358] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6077.068763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 6085.044396] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6093.287579] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6118.374747] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6118.382990] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6118.389124] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6118.397100] Lustre: Skipped 1 previous similar message [ 6118.415949] Lustre: Skipped 1 previous similar message [ 6120.601981] Lustre: server umount lustre-OST0000 complete [ 6128.617889] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6128.633334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6131.017405] Lustre: server umount lustre-MDT0000 complete [ 6133.733511] LustreError: 184584:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6133.760101] LustreError: 184584:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6134.843884] LustreError: 184563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976100 with bad export cookie 14551648851250537563 [ 6134.850636] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6134.860883] LustreError: 184563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6135.203596] Lustre: server umount lustre-MDT0001 complete [ 6146.588792] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 6149.607614] Key type lgssc unregistered [ 6149.843455] LNet: 188142:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6149.861244] LNetError: 188142:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6149.892696] LNet: Removed LNI 192.168.201.106@tcp [ 6151.001095] Key type .llcrypt unregistered [ 6151.002717] Key type ._llcrypt unregistered [ 6181.263672] Key type ._llcrypt registered [ 6181.265801] Key type .llcrypt registered [ 6181.363325] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6182.562876] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6182.663026] alg: No test for adler32 (adler32-zlib) [ 6183.774650] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 6184.209244] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 6185.944986] Key type lgssc registered [ 6187.425224] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6196.718293] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6196.767896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6198.403619] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6203.732391] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6208.999490] LustreError: 189571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6209.036621] LustreError: 189571:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6214.113704] LustreError: 189572:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6216.709977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6217.323379] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6221.967946] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6229.445911] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6235.238804] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6242.804280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6243.333539] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6244.347093] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6249.634721] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6258.430356] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6269.922379] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6269.940286] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6269.970325] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6271.971451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6271.972572] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6271.974114] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6275.758728] Lustre: server umount lustre-OST0000 complete [ 6287.868281] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffffa08b053be800.ost_server_uuid 50 [ 6291.091639] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6309.352130] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6309.371044] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6309.407958] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6309.421028] Lustre: Skipped 2 previous similar messages [ 6311.468707] LustreError: 190300:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6311.469609] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6311.618904] Lustre: server umount lustre-MDT0000 complete [ 6316.513176] LustreError: 189571:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6316.555507] LustreError: 189571:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6316.936116] LustreError: 189552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976282 with bad export cookie 15055017983041544207 [ 6316.949922] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6316.956702] LustreError: 189552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6317.509404] Lustre: server umount lustre-MDT0001 complete [ 6329.300525] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 13:51:32 (1788976292) [ 6337.861442] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6338.532054] LustreError: 193500:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6338.631262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6344.106638] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6348.771405] LustreError: 193501:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6348.782215] LustreError: 193501:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 6352.933577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6353.272948] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6357.692446] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6365.651589] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6371.537411] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6378.349801] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6378.696908] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6383.868524] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6384.584104] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6392.414385] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6399.222715] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6399.513743] Lustre: lustre-OST0001: new disk, initializing [ 6399.519831] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6399.527309] Lustre: 196340:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6399.624241] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6406.490761] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6408.843719] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6408.859038] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6408.912719] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6416.662849] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6419.438404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6419.458587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6419.464112] Lustre: Skipped 1 previous similar message [ 6424.551493] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6424.574506] Lustre: Skipped 2 previous similar messages [ 6425.707245] Lustre: server umount lustre-OST0000 complete [ 6427.654689] LustreError: 195321:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6433.435389] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6433.673385] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6439.650880] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6440.012141] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6455.269054] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6455.288748] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6455.299375] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6455.316311] Lustre: Skipped 2 previous similar messages [ 6455.326440] Lustre: Skipped 1 previous similar message [ 6460.405466] LustreError: 195320:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6460.423345] LustreError: 195320:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 17 previous similar messages [ 6461.363636] Lustre: server umount lustre-OST0001 complete [ 6473.535509] Lustre: server umount lustre-MDT0000 complete [ 6477.744955] LustreError: 193482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976443 with bad export cookie 15055017983041545278 [ 6477.753729] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6477.768715] LustreError: 193482:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6478.079603] Lustre: server umount lustre-MDT0001 complete [ 6505.614856] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 13:54:29 (1788976469) [ 6514.181346] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6514.686687] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6514.754740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6515.160489] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6522.927476] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6534.645810] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6534.777424] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6534.833621] Lustre: 199784:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6540.498712] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6548.184794] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6554.700854] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6561.690717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6561.959491] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6568.110643] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6568.942289] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6577.391333] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6584.694684] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6584.818657] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6585.093115] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6585.098377] Lustre: Skipped 2 previous similar messages [ 6592.046983] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6601.534380] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6607.894263] Lustre: lustre-OST0000: Not available for connect from 192.168.201.6@tcp (stopping) [ 6607.901505] Lustre: Skipped 6 previous similar messages [ 6608.359420] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6608.373814] Lustre: Skipped 2 previous similar messages [ 6619.104203] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6619.291988] Lustre: server umount lustre-OST0000 complete [ 6623.227054] LustreError: 201635:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6623.254794] LustreError: 201635:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 6639.079026] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6639.088552] LustreError: Skipped 2 previous similar messages [ 6642.000434] Lustre: server umount lustre-OST0001 complete [ 6647.803419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6647.809253] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6647.832142] Lustre: Skipped 3 previous similar messages [ 6647.847672] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6647.857855] Lustre: Skipped 10 previous similar messages [ 6652.669846] Lustre: server umount lustre-MDT0000 complete [ 6657.799579] LustreError: 199777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976623 with bad export cookie 15055017983041546685 [ 6657.805434] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6657.821440] LustreError: 199777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6658.195511] Lustre: server umount lustre-MDT0001 complete [ 6688.611319] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 13:57:31 (1788976651) [ 6707.805630] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 6719.199985] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 6765.572977] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6766.011357] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6766.098901] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6766.340672] Lustre: lustre-MDT0000: new disk, initializing [ 6766.603675] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6766.637393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6773.306484] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6785.508517] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6785.807714] Lustre: 207884:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6785.900129] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6785.912125] Lustre: Skipped 1 previous similar message [ 6786.083446] Lustre: lustre-MDT0001: new disk, initializing [ 6786.297321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6786.313296] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6793.409878] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6809.011831] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6816.271464] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6825.529716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6825.806560] Lustre: lustre-OST0000: new disk, initializing [ 6825.814342] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6825.819694] Lustre: 209858:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6831.915360] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6833.781888] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6833.790484] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6833.890055] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6841.865704] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6846.922616] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6847.133493] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6851.942289] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6852.160944] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6854.625699] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6854.630702] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6854.638659] LustreError: Skipped 1 previous similar message [ 6854.641212] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6854.655992] Lustre: Skipped 2 previous similar messages [ 6854.670727] Lustre: Skipped 2 previous similar messages [ 6860.303653] Lustre: server umount lustre-OST0000 complete [ 6870.960017] Lustre: server umount lustre-MDT0000 complete [ 6874.895906] LustreError: 208681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788976840 with bad export cookie 15055017983041548253 [ 6874.899719] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6874.905408] LustreError: 208681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6875.192289] Lustre: server umount lustre-MDT0001 complete [ 6884.489185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6884.905977] LustreError: 212075:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6884.919208] LustreError: 212075:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [ 6890.585270] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6901.178498] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6905.875307] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6915.174948] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6921.961638] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6928.510363] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6936.114515] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 6943.967533] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6949.723804] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6950.065254] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6955.846633] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6956.071788] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6960.616302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6972.386190] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6972.578858] Lustre: server umount lustre-OST0000 complete [ 6979.979054] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6984.455574] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7013.584391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7020.465449] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7027.264698] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7027.282217] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7027.286455] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7027.316972] Lustre: Skipped 1 previous similar message [ 7028.248419] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7033.058550] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 3 sec [ 7037.921936] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7037.936048] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7037.936940] LustreError: Skipped 2 previous similar messages [ 7037.946046] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7037.946068] Lustre: Skipped 13 previous similar messages [ 7037.985254] Lustre: Skipped 5 previous similar messages [ 7043.743864] Lustre: server umount lustre-OST0000 complete [ 7054.325892] Lustre: server umount lustre-MDT0000 complete [ 7058.308757] LustreError: 212056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977024 with bad export cookie 15055017983041549135 [ 7058.318542] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7058.327418] LustreError: 212056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7058.851090] Lustre: server umount lustre-MDT0001 complete [ 7068.776767] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 14:03:52 (1788977032) [ 7076.337191] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7076.931633] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7076.935343] Lustre: Skipped 6 previous similar messages [ 7081.925752] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7092.197659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7099.771822] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7110.290291] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7118.943821] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7126.426532] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7132.414680] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7140.006330] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7144.810712] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7148.212701] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 3 sec [ 7153.362497] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7153.614736] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7161.959847] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7162.246139] Lustre: lustre-OST0001: new disk, initializing [ 7162.263298] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7162.278561] Lustre: 221449:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7169.688098] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7170.097072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7170.105580] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7170.143958] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7181.462357] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7187.309671] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7187.407233] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7193.233483] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7193.695544] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7200.741847] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7200.761921] LustreError: Skipped 1 previous similar message [ 7202.084521] Lustre: server umount lustre-OST0001 complete [ 7205.865837] LustreError: 219877:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7205.891438] LustreError: 219877:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [ 7208.248286] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7208.545487] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 7214.182741] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7214.487078] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 7244.634578] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7250.418882] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7250.458643] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 7250.479024] Lustre: lustre-OST0001-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7250.494420] Lustre: Skipped 1 previous similar message [ 7252.780217] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7262.705458] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7264.272760] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7272.649649] Lustre: server umount lustre-OST0001 complete [ 7296.473927] Lustre: server umount lustre-OST0000 complete [ 7301.606652] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7301.617620] Lustre: Skipped 8 previous similar messages [ 7301.624858] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7301.638615] Lustre: Skipped 15 previous similar messages [ 7307.095538] Lustre: server umount lustre-MDT0000 complete [ 7311.675914] LustreError: 218035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977277 with bad export cookie 15055017983041550101 [ 7311.686599] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7311.701474] LustreError: 218035:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7318.409958] Lustre: server umount lustre-MDT0001 complete [ 7346.631935] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 14:08:30 (1788977310) [ 7354.953758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7355.279372] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7355.306215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7360.467974] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7369.712056] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7369.850823] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7369.924104] Lustre: 226710:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7374.530693] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7382.302513] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7388.558465] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7395.593924] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7395.883686] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7403.351359] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7411.493632] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7423.084023] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7423.163033] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7429.251828] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7437.398436] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7439.100841] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7445.425840] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 7445.705087] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7450.945950] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 7451.138809] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7456.804557] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffffa08b209bc000.ost_server_uuid 50 [ 7458.200576] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffffa08b209bc000.ost_server_uuid in IDLE state after 0 sec [ 7459.271965] Lustre: Permanently deactivating lustre-OST0001 [ 7459.294996] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 7463.241837] Lustre: Permanently reactivating lustre-OST0001 [ 7468.513675] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7468.526060] LustreError: Skipped 3 previous similar messages [ 7470.174697] Lustre: server umount lustre-OST0001 complete [ 7480.710743] Lustre: server umount lustre-OST0000 complete [ 7491.459750] Lustre: server umount lustre-MDT0000 complete [ 7495.907812] LustreError: 226704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977461 with bad export cookie 15055017983041551389 [ 7495.909787] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7495.927411] LustreError: 226704:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7496.344197] Lustre: server umount lustre-MDT0001 complete [ 7522.829810] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 14:11:26 (1788977486) [ 7531.274597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7531.770274] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7531.835710] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7531.851788] Lustre: Skipped 5 previous similar messages [ 7536.976937] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7546.040204] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7546.150904] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7546.190461] Lustre: 233406:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7551.392328] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7560.280420] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7567.145529] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7575.507226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7575.831714] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7575.869867] Lustre: Permanently deactivating lustre-OST0000 [ 7575.900269] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7583.806258] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7583.819845] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7593.131035] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7601.813565] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7601.892634] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 7602.209431] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7602.224812] Lustre: Skipped 11 previous similar messages [ 7609.650845] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7610.442887] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7619.951845] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7621.870260] Lustre: Permanently reactivating lustre-OST0000 [ 7621.884897] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7621.893104] Lustre: Skipped 2 previous similar messages [ 7629.411611] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7629.413068] Lustre: lustre-OST0000: haven't heard from client lustre-MDT0000-mdtlov_UUID (at 0@lo) in 46 seconds. I think it's dead, and I am evicting it. exp 00000000be95ab33, cur 1788977595 deadline 1788977579 last 1788977549 [ 7629.426916] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7629.470036] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7629.557710] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7645.702898] Lustre: server umount lustre-OST0001 complete [ 7656.185861] Lustre: server umount lustre-OST0000 complete [ 7674.336143] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7674.533742] Lustre: server umount lustre-MDT0000 complete [ 7678.210254] LustreError: 233398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977643 with bad export cookie 15055017983041553062 [ 7678.213711] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7678.226960] LustreError: 233398:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7678.627110] Lustre: server umount lustre-MDT0001 complete [ 7690.466101] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 7693.324701] Key type lgssc unregistered [ 7693.659266] LNet: 238291:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7693.662727] LNetError: 238291:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7693.680673] LNet: Removed LNI 192.168.201.106@tcp [ 7694.531664] Key type .llcrypt unregistered [ 7694.533152] Key type ._llcrypt unregistered [ 7720.112764] Key type ._llcrypt registered [ 7720.116784] Key type .llcrypt registered [ 7720.192247] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7721.621498] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7721.683450] alg: No test for adler32 (adler32-zlib) [ 7722.789807] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 7723.081084] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 7724.792816] Key type lgssc registered [ 7726.051764] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7735.736545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7735.752981] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7737.285735] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7741.726519] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7746.551369] LustreError: 239757:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7746.586865] LustreError: 239757:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 7751.658114] LustreError: 239756:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7751.998158] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7752.278954] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7752.313680] Lustre: 239745:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7752.327968] Lustre: Permanently deactivating lustre-MDT0001 [ 7752.342430] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7752.578313] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7757.729739] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7760.919800] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7765.671483] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7772.472183] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7781.887963] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7782.308154] LustreError: 241598:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7782.338750] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7787.503501] LustreError: 241599:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7787.556600] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7790.057907] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7792.623200] LustreError: 241853:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7797.769737] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 7799.065936] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7802.854272] LustreError: 241598:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7802.884280] LustreError: 241598:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 7807.027836] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7807.484395] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7812.617608] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7812.622119] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7814.943565] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7823.880266] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7827.850218] Lustre: Permanently reactivating lustre-MDT0001 [ 7827.884981] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7827.897905] Lustre: Skipped 2 previous similar messages [ 7830.002353] Lustre: lustre-MDT0001: Client 91c42c9c-121b-47e3-a2c4-5893f80fb215 (at 192.168.201.6@tcp) reconnecting [ 7834.783541] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7834.797139] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 7834.804819] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 7834.826366] LustreError: 243470:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 7834.830753] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 7834.840392] LustreError: 243470:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 7834.843864] LustreError: 243470:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.201.106@tcp: cfg command failed: rc = -108 [ 7834.843891] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 7834.843891] [ 7834.850131] LustreError: 239748:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 7836.882605] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7838.797747] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7844.959693] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7845.244463] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7849.894552] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7850.095954] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7852.973204] Lustre: Permanently deactivating lustre-MDT0001 [ 7852.993352] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7852.999657] Lustre: Skipped 2 previous similar messages [ 7861.921256] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7870.433313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7870.445296] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7870.463486] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7870.964227] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7870.976091] Lustre: Skipped 1 previous similar message [ 7876.072440] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7876.087153] Lustre: Skipped 4 previous similar messages [ 7876.888793] Lustre: server umount lustre-MDT0000 complete [ 7880.814945] LustreError: 239737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788977846 with bad export cookie 7572301248496064589 [ 7880.821351] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7880.826342] LustreError: 239737:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7881.185293] LustreError: 242218:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7881.205516] LustreError: 242218:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 7881.323446] Lustre: server umount lustre-MDT0001 complete [ 7887.096144] Lustre: server umount lustre-OST0000 complete [ 7900.833627] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 14:17:44 (1788977864) [ 7913.895184] Lustre: server umount lustre-OST0001 complete [ 7921.970896] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 7932.802854] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 7982.318853] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7982.630558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7982.640250] Lustre: Skipped 2 previous similar messages [ 7982.671542] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7982.790622] Lustre: lustre-MDT0000: new disk, initializing [ 7982.849971] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7982.872093] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7987.144037] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 7997.555622] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7997.658882] Lustre: 248041:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7997.705353] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7997.708295] Lustre: Skipped 1 previous similar message [ 7997.771484] Lustre: lustre-MDT0001: new disk, initializing [ 7997.844462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7997.906249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7997.923480] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8002.018597] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8011.922342] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8018.425189] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8025.599563] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8025.875323] Lustre: lustre-OST0000: new disk, initializing [ 8025.879437] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8025.888199] Lustre: 250007:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8026.035367] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8032.296822] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8032.843451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8032.851721] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8032.963762] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8042.440681] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8044.915345] LustreError: 248049:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8053.147395] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8053.314244] Lustre: lustre-OST0001: new disk, initializing [ 8053.317809] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8053.325571] Lustre: 251122:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8054.985199] LustreError: 248049:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8055.009893] LustreError: 248048:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 8059.540175] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8063.092136] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8063.099083] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8063.217835] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8065.040065] LustreError: 248048:0:(mdt_handler.c:3274:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 8068.814789] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8070.564913] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 8073.710957] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8073.719029] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8073.735671] Lustre: Skipped 4 previous similar messages [ 8073.741092] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8078.821891] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8078.832518] Lustre: Skipped 2 previous similar messages [ 8087.008157] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8087.264331] Lustre: server umount lustre-OST0001 complete [ 8089.071208] LustreError: 249991:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8089.084534] LustreError: 249991:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8104.929379] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8104.933813] LustreError: Skipped 1 previous similar message [ 8104.937092] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8104.945446] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8104.958676] Lustre: Skipped 1 previous similar message [ 8104.968127] Lustre: Skipped 3 previous similar messages [ 8110.944146] Lustre: server umount lustre-OST0000 complete [ 8115.681718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8115.691233] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8121.487808] Lustre: server umount lustre-MDT0000 complete [ 8125.514438] LustreError: 248033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978091 with bad export cookie 7572301248496067396 [ 8125.516695] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8125.535253] LustreError: 248033:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8132.093536] Lustre: server umount lustre-MDT0001 complete [ 8143.782836] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8146.612873] Key type lgssc unregistered [ 8146.920914] LNet: 253158:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8146.939708] LNetError: 253158:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8146.962830] LNet: Removed LNI 192.168.201.106@tcp [ 8147.824605] Key type .llcrypt unregistered [ 8147.827106] Key type ._llcrypt unregistered [ 8188.184414] Key type ._llcrypt registered [ 8188.187296] Key type .llcrypt registered [ 8188.269419] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8189.712994] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8189.805810] alg: No test for adler32 (adler32-zlib) [ 8190.921463] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 8191.110573] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8192.768937] Key type lgssc registered [ 8194.320370] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8202.124700] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8202.146951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8203.506149] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8203.522603] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8203.749951] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8209.022127] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8219.068508] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8219.230291] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8219.268788] Lustre: 255362:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8219.562315] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8224.382313] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8232.436481] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8239.045213] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8246.869736] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8247.207125] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8247.587639] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8254.338483] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8255.186655] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 8262.822769] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8316.389989] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8316.413244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8316.438733] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8316.898729] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8322.025129] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8322.033987] Lustre: Skipped 2 previous similar messages [ 8322.328873] Lustre: server umount lustre-OST0000 complete [ 8325.999551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 8337.720913] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8338.176809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8344.432111] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8346.356780] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8346.377971] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 8346.390569] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 8346.397394] Lustre: Skipped 1 previous similar message [ 8352.820033] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8392.682199] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8392.688181] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8392.717033] Lustre: Skipped 2 previous similar messages [ 8392.723227] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8397.800632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8397.809836] Lustre: Skipped 2 previous similar messages [ 8398.375233] Lustre: server umount lustre-OST0000 complete [ 8402.914855] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8402.929458] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8408.893307] Lustre: server umount lustre-MDT0000 complete [ 8413.320685] LustreError: 255356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978379 with bad export cookie 5562762038375698439 [ 8413.336289] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8413.338468] LustreError: 255356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8413.725626] Lustre: server umount lustre-MDT0001 complete [ 8427.546870] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8431.689367] Key type lgssc unregistered [ 8432.016648] LNet: 260785:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8432.039236] LNetError: 260785:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8432.075911] LNet: Removed LNI 192.168.201.106@tcp [ 8433.265843] Key type .llcrypt unregistered [ 8433.272151] Key type ._llcrypt unregistered [ 8464.853291] Key type ._llcrypt registered [ 8464.860220] Key type .llcrypt registered [ 8465.026708] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8465.945794] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8465.975221] alg: No test for adler32 (adler32-zlib) [ 8467.115401] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 8467.425155] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8469.136414] Key type lgssc registered [ 8470.274396] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8477.773491] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8477.805119] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8479.187049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8483.921447] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8488.419366] LustreError: 262196:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8488.442449] LustreError: 262196:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8492.717537] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8492.952480] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8497.912638] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8506.667326] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8513.602751] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8522.218731] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8522.595235] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8524.691822] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 8530.224582] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8538.907877] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8563.701469] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8563.705073] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8563.714079] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8563.719324] Lustre: Skipped 1 previous similar message [ 8567.383434] Lustre: server umount lustre-OST0000 complete [ 8573.922457] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8573.937190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8573.949217] Lustre: Skipped 2 previous similar messages [ 8577.722929] Lustre: server umount lustre-MDT0000 complete [ 8579.044059] LustreError: 262197:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8579.071989] LustreError: 262197:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8581.540201] LustreError: 262179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978547 with bad export cookie 12121726451989167639 [ 8581.546048] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8581.577037] LustreError: 262179:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8581.999969] Lustre: server umount lustre-MDT0001 complete [ 8594.566370] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8597.331594] Key type lgssc unregistered [ 8597.610740] LNet: 266505:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8597.621834] LNetError: 266505:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8597.645210] LNet: Removed LNI 192.168.201.106@tcp [ 8598.634598] Key type .llcrypt unregistered [ 8598.639116] Key type ._llcrypt unregistered [ 8617.096881] Key type ._llcrypt registered [ 8617.098272] Key type .llcrypt registered [ 8617.271292] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8618.658051] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8618.756858] alg: No test for adler32 (adler32-zlib) [ 8619.886817] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 8620.291145] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8622.040308] Key type lgssc registered [ 8623.023486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8631.589671] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8631.611464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8633.168510] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8639.301100] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8642.532515] LustreError: 267722:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8642.569834] LustreError: 267722:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8647.651421] LustreError: 267723:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8650.172046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8650.661462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8655.641107] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8664.099532] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8670.216585] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8677.906254] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8678.227853] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8684.439348] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8692.915066] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8702.727720] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 8717.288165] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8717.307585] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8717.337565] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8718.315050] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8718.317327] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8718.318334] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8718.347748] Lustre: Skipped 1 previous similar message [ 8720.018504] Lustre: server umount lustre-OST0000 complete [ 8727.525734] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8727.545454] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8727.580420] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8731.034785] Lustre: server umount lustre-MDT0000 complete [ 8733.675829] LustreError: 267722:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8733.700856] LustreError: 267722:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8735.393099] LustreError: 267703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978701 with bad export cookie 13516058522211370495 [ 8735.418812] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8735.422874] LustreError: 267703:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8735.939835] Lustre: server umount lustre-MDT0001 complete [ 8749.495146] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8753.346914] Key type lgssc unregistered [ 8753.836813] LNet: 271422:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8753.842556] LNetError: 271422:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8753.858296] LNet: Removed LNI 192.168.201.106@tcp [ 8754.804920] Key type .llcrypt unregistered [ 8754.808951] Key type ._llcrypt unregistered [ 8784.318854] Key type ._llcrypt registered [ 8784.321098] Key type .llcrypt registered [ 8784.425614] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8785.588915] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8785.794749] alg: No test for adler32 (adler32-zlib) [ 8787.003776] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 8787.364189] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8789.025366] Key type lgssc registered [ 8790.414564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8799.824770] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8799.845605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8801.515984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8806.446680] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8812.004784] LustreError: 272837:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8812.013267] LustreError: 272837:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8815.962481] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8816.421199] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8821.873312] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8830.378465] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8836.713749] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8844.326341] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8844.765657] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8846.856540] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 8851.218658] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8861.019484] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8891.876846] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8891.895526] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8891.896840] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8891.910332] Lustre: Skipped 1 previous similar message [ 8896.253965] Lustre: server umount lustre-OST0000 complete [ 8902.114503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8902.117610] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8902.135691] Lustre: Skipped 2 previous similar messages [ 8902.143624] Lustre: Skipped 1 previous similar message [ 8906.103651] Lustre: server umount lustre-MDT0000 complete [ 8907.240513] LustreError: 272836:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8907.284935] LustreError: 272836:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 8910.084207] LustreError: 272818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788978875 with bad export cookie 16291173024185278337 [ 8910.095097] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8910.105260] LustreError: 272818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8910.500685] Lustre: server umount lustre-MDT0001 complete [ 8924.202506] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 8927.991942] Key type lgssc unregistered [ 8928.377872] LNet: 277144:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8928.382531] LNetError: 277144:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8928.432837] LNet: Removed LNI 192.168.201.106@tcp [ 8929.889745] Key type .llcrypt unregistered [ 8929.893480] Key type ._llcrypt unregistered [ 8952.720457] Key type ._llcrypt registered [ 8952.725806] Key type .llcrypt registered [ 8952.882607] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 8954.270714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8954.301307] alg: No test for adler32 (adler32-zlib) [ 8955.481472] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 8955.790606] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 8957.464230] Key type lgssc registered [ 8959.073849] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8967.578038] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8967.592776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8969.022747] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8974.050148] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8978.415873] LustreError: 278379:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8978.436049] LustreError: 278379:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [ 8983.529442] LustreError: 278381:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8983.654608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8983.960130] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8989.264881] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 8996.426620] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9004.899079] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9013.188853] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9013.607878] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9014.646136] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 9020.973828] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9030.068663] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9044.452499] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9044.458514] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9044.479397] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9044.505922] Lustre: Skipped 1 previous similar message [ 9044.539043] Lustre: Skipped 1 previous similar message [ 9048.544120] Lustre: server umount lustre-OST0000 complete [ 9054.705632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9054.718212] Lustre: Skipped 1 previous similar message [ 9054.720108] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9054.733514] Lustre: Skipped 1 previous similar message [ 9059.748552] Lustre: server umount lustre-MDT0000 complete [ 9059.813550] LustreError: 278370:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9059.852100] LustreError: 278370:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9063.950340] LustreError: 278344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788979029 with bad export cookie 10292825320929279268 [ 9063.962470] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9063.967349] LustreError: 278344:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9064.554444] Lustre: server umount lustre-MDT0001 complete [ 9077.667804] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9080.730964] Key type lgssc unregistered [ 9081.025248] LNet: 282079:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9081.036968] LNetError: 282079:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9081.054515] LNet: Removed LNI 192.168.201.106@tcp [ 9081.848338] Key type .llcrypt unregistered [ 9081.852226] Key type ._llcrypt unregistered [ 9099.273220] Key type ._llcrypt registered [ 9099.276370] Key type .llcrypt registered [ 9099.567989] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 9121.423771] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9139.184486] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9140.560432] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9140.736396] alg: No test for adler32 (adler32-zlib) [ 9142.025458] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 9142.395168] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 9144.096443] Key type lgssc registered [ 9145.681970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9191.120354] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9191.158528] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9192.539813] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9192.573714] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9192.678329] Lustre: lustre-MDT0000: new disk, initializing [ 9192.797426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9192.821811] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9198.263727] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9211.439886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9211.619685] Lustre: 285635:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9211.676108] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9211.680196] Lustre: Skipped 1 previous similar message [ 9211.752833] Lustre: lustre-MDT0001: new disk, initializing [ 9211.896750] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9211.929072] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9211.954472] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9217.667136] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9229.091472] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9234.951822] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9244.653646] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9245.131554] Lustre: lustre-OST0000: new disk, initializing [ 9245.133709] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9245.161925] Lustre: 287604:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9245.340418] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9253.967274] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9254.486740] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9254.502667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9254.719073] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9265.905140] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9272.369423] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9272.829711] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9279.082634] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9279.376139] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9285.604630] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9285.606696] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9285.622898] Lustre: Skipped 1 previous similar message [ 9290.743679] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9290.761490] Lustre: Skipped 2 previous similar messages [ 9295.844198] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9295.860474] Lustre: Skipped 1 previous similar message [ 9296.352301] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9296.661767] Lustre: server umount lustre-OST0000 complete [ 9300.960941] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9300.970844] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9300.982790] Lustre: Skipped 1 previous similar message [ 9306.081104] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9306.086217] Lustre: Skipped 1 previous similar message [ 9315.298162] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9315.555049] Lustre: server umount lustre-MDT0000 complete [ 9319.549191] LustreError: 285627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788979285 with bad export cookie 14681249136792614422 [ 9319.550749] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9319.565042] LustreError: 285627:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9319.928712] Lustre: server umount lustre-MDT0001 complete [ 9329.264809] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 14:41:32 (1788979292) [ 9337.289507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9337.781556] LustreError: 290019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9337.801053] LustreError: 290019:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [ 9337.873504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9342.955297] LustreError: 290020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9343.075778] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9348.070543] LustreError: 290019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9353.109761] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9353.189794] LustreError: 290020:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9353.538594] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9358.380650] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9367.108214] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9373.808724] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9382.221386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9382.803140] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9390.594982] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9399.349695] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9422.817957] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9422.835518] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9422.854308] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9422.866415] Lustre: Skipped 2 previous similar messages [ 9422.879676] Lustre: Skipped 3 previous similar messages [ 9429.116773] Lustre: server umount lustre-OST0000 complete [ 9435.617726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9435.623286] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9440.530832] Lustre: server umount lustre-MDT0000 complete [ 9443.302106] LustreError: 290019:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9443.329315] LustreError: 290019:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [ 9444.656442] LustreError: 290001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788979410 with bad export cookie 14681249136792615304 [ 9444.680793] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9445.133491] Lustre: server umount lustre-MDT0001 complete [ 9457.483478] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [ 9461.567552] Key type lgssc unregistered [ 9461.954487] LNet: 293580:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9461.961111] LNetError: 293580:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9463.016531] LNet: Removed LNI 192.168.201.106@tcp [ 9464.257239] Key type .llcrypt unregistered [ 9464.262626] Key type ._llcrypt unregistered [ 9496.787363] Key type ._llcrypt registered [ 9496.789524] Key type .llcrypt registered [ 9496.884870] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9514.066268] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9515.460734] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9515.475337] alg: No test for adler32 (adler32-zlib) [ 9516.580923] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [ 9516.862976] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [ 9518.609894] Key type lgssc registered [ 9520.083777] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9572.475766] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9572.503464] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9573.723191] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9573.747959] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9573.825683] Lustre: lustre-MDT0000: new disk, initializing [ 9573.904111] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9573.919286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9579.270363] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9590.900141] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9591.049496] Lustre: 297407:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9591.092423] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9591.095320] Lustre: Skipped 1 previous similar message [ 9591.154709] Lustre: lustre-MDT0001: new disk, initializing [ 9591.305170] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9591.354649] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9591.377193] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9596.587472] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9607.687991] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9615.681582] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9623.760589] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9624.309867] Lustre: lustre-OST2710: new disk, initializing [ 9624.324253] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 9624.345315] Lustre: 299375:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9624.506408] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 9632.164654] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9632.793925] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 9632.803698] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 9632.880750] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 9645.688716] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9657.862744] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9658.190580] Lustre: lustre-OST03e8: new disk, initializing [ 9658.207495] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 9658.235165] Lustre: 300444:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9658.385191] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 9665.917099] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9668.294978] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 9668.306500] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 9668.384387] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 9674.979681] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9682.994691] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 9683.223508] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9688.092862] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 9688.352688] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9693.564563] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 9693.821732] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9699.727345] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 9699.955574] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9705.928559] Lustre: server umount lustre-MDT0000 complete [ 9709.029193] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9709.044728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9709.554243] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9710.240046] LustreError: 299374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788979675 with bad export cookie 8589570250961408294 [ 9710.246344] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9710.261270] LustreError: 299374:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9710.710241] Lustre: server umount lustre-MDT0001 complete [ 9725.675882] Lustre: server umount lustre-OST2710 complete [ 9740.902343] Lustre: server umount lustre-OST03e8 complete [ 9762.690268] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [ 9776.530444] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9824.827096] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9825.329457] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9825.383829] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9825.484674] Lustre: lustre-MDT0000: new disk, initializing [ 9825.613908] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9825.633248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9831.169072] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9843.332608] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9843.411743] Lustre: 305697:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9843.455951] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9843.459108] Lustre: Skipped 1 previous similar message [ 9843.519910] Lustre: lustre-MDT0001: new disk, initializing [ 9843.600993] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9843.650932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9843.667886] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9849.303717] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9860.472558] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9867.473994] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9875.314506] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9875.563652] Lustre: lustre-OST0000: new disk, initializing [ 9875.567068] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9875.574200] Lustre: 307665:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9875.710429] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9881.773823] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9881.789166] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9881.950274] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9883.644722] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9893.906065] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9899.366965] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9899.652890] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9904.732716] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9905.078258] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9907.685713] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9907.709547] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9907.716499] Lustre: Skipped 3 previous similar messages [ 9907.720835] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9907.747730] Lustre: Skipped 1 previous similar message [ 9912.802430] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9912.814950] Lustre: Skipped 1 previous similar message [ 9913.895216] Lustre: server umount lustre-OST0000 complete [ 9920.482285] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9920.490404] LustreError: Skipped 1 previous similar message [ 9920.499717] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9920.529809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9923.044280] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9923.066477] Lustre: Skipped 1 previous similar message [ 9925.191444] Lustre: server umount lustre-MDT0000 complete [ 9928.165757] LustreError: 305708:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9928.206836] LustreError: 305708:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 6 previous similar messages [ 9929.600524] LustreError: 305688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788979895 with bad export cookie 8589570250961409974 [ 9929.601745] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9929.613331] LustreError: 305688:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9930.013620] Lustre: server umount lustre-MDT0001 complete [ 9941.835800] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 14:51:44 (1788979904) [ 9944.074933] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 9946.796496] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 14:51:49 (1788979909) [ 9951.506958] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [ 9981.227109] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9981.583665] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9981.619973] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9981.868128] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9987.827527] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [ 9998.242965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9998.396036] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9998.454241] Lustre: 311527:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10004.123675] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10028.149846] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10028.422231] Lustre: Denying initial registration attempt from nid 192.168.201.106@tcp, specified as failover [10028.430559] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [10028.443464] LustreError: 312982:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -99 [10028.458516] LustreError: 312982:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10028.471353] LustreError: 312982:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10028.594507] Lustre: server umount lustre-OST0000 complete [10028.602151] LustreError: 312982:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [10037.734188] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10037.752639] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10037.755405] Lustre: Skipped 2 previous similar messages [10037.800018] Lustre: Skipped 1 previous similar message [10042.852428] LustreError: 311535:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10042.870379] LustreError: 311535:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10042.948065] Lustre: server umount lustre-MDT0000 complete [10047.350246] LustreError: 311521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980013 with bad export cookie 8589570250961410849 [10047.355693] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10047.364228] LustreError: 311521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10047.880265] Lustre: server umount lustre-MDT0001 complete [10062.192815] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [10064.857063] Key type lgssc unregistered [10065.207251] LNet: 314103:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10065.215841] LNetError: 314103:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10065.241621] LNet: Removed LNI 192.168.201.106@tcp [10066.185927] Key type .llcrypt unregistered [10066.188238] Key type ._llcrypt unregistered [10079.044420] Key type ._llcrypt registered [10079.046456] Key type .llcrypt registered [10079.123021] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10080.033718] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10080.181668] alg: No test for adler32 (adler32-zlib) [10081.266152] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [10081.572347] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [10083.272284] Key type lgssc registered [10084.463997] Lustre: Echo OBD driver; http://www.lustre.org/ [10132.975181] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10142.397365] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10142.414487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10143.699318] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10143.725481] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10143.901593] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10148.335461] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10157.951820] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10158.121042] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10158.160667] Lustre: 316862:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10158.368417] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10164.339678] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10172.743755] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10179.321940] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10200.062571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10200.398120] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10200.674457] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10208.733319] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10218.363780] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10222.051033] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10222.071877] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10222.094373] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10222.571068] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10227.704190] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10227.724338] Lustre: Skipped 1 previous similar message [10232.819625] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10232.830189] Lustre: Skipped 1 previous similar message [10234.848411] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10235.074950] Lustre: server umount lustre-OST0000 complete [10240.482041] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10240.485667] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10240.493061] Lustre: Skipped 1 previous similar message [10240.500313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10243.044535] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10246.094749] Lustre: server umount lustre-MDT0000 complete [10250.544413] LustreError: 316854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980216 with bad export cookie 5840802871891207367 [10250.557666] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10250.563516] LustreError: 316854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10251.154772] Lustre: server umount lustre-MDT0001 complete [10263.684177] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [10266.702511] Key type lgssc unregistered [10267.055672] LNet: 320438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10267.070899] LNetError: 320438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10268.141100] LNet: Removed LNI 192.168.201.106@tcp [10269.022377] Key type .llcrypt unregistered [10269.024680] Key type ._llcrypt unregistered [10300.907534] Key type ._llcrypt registered [10300.916976] Key type .llcrypt registered [10301.045027] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10302.480266] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10302.502114] alg: No test for adler32 (adler32-zlib) [10303.548418] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [10303.778703] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [10305.520235] Key type lgssc registered [10307.159773] Lustre: Echo OBD driver; http://www.lustre.org/ [10315.481868] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10315.508070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10317.068689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10322.272911] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10326.498182] LustreError: 321853:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10326.524229] LustreError: 321853:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10331.625364] LustreError: 321854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10332.489587] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10332.957599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10338.028540] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10347.729520] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10355.312867] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10365.239651] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10365.877283] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10374.266498] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10385.565951] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10396.647992] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10396.653700] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10396.677063] Lustre: Skipped 2 previous similar messages [10396.701378] Lustre: Skipped 2 previous similar messages [10399.276733] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [10401.765970] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10401.793563] Lustre: Skipped 2 previous similar messages [10402.873759] Lustre: server umount lustre-MDT0000 complete [10404.346874] LustreError: 321848:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10406.881829] LustreError: 321854:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10406.905510] LustreError: 321854:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10408.181770] LustreError: 321835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980373 with bad export cookie 9903185639595513441 [10408.186567] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10408.190039] LustreError: 321835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10409.527908] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [10412.001658] LustreError: 324159:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10412.002511] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10412.012079] LustreError: 324159:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10414.053369] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10414.066912] Lustre: Skipped 1 previous similar message [10414.894623] Lustre: server umount lustre-MDT0001 complete [10422.337658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [10432.188858] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10432.494412] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x896f2edeb0d73245 [10432.500659] Lustre: MGC192.168.201.106@tcp: Connection restored to 0@lo (at 0@lo) [10432.901883] LustreError: 323685:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10432.927898] LustreError: 323685:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [10433.043867] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10433.156571] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [10438.352321] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10443.253866] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10448.250977] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10448.355712] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [10448.360855] Lustre: Skipped 1 previous similar message [10448.535709] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10453.863340] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10454.022221] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [10463.334960] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10465.245748] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [10473.197958] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10475.137604] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [10484.195624] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10484.198525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10484.208692] LustreError: Skipped 1 previous similar message [10484.213087] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10484.229267] Lustre: Skipped 1 previous similar message [10484.255784] Lustre: Skipped 1 previous similar message [10487.343559] Lustre: server umount lustre-OST0000 complete [10494.434313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10494.444428] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10506.215075] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10506.598751] Lustre: server umount lustre-MDT0000 complete [10510.335052] LustreError: 325448:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10510.357880] LustreError: 325448:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 10 previous similar messages [10511.461867] LustreError: 321835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980477 with bad export cookie 9903185639595520581 [10511.477182] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10511.479864] LustreError: 321835:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10511.962174] Lustre: server umount lustre-MDT0001 complete [10525.631599] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [10528.384319] Key type lgssc unregistered [10528.654807] LNet: 328136:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10528.661452] LNetError: 328136:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10529.703885] LNet: Removed LNI 192.168.201.106@tcp [10530.779688] Key type .llcrypt unregistered [10530.782048] Key type ._llcrypt unregistered [10560.803928] Key type ._llcrypt registered [10560.808769] Key type .llcrypt registered [10560.940522] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10561.733683] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10561.784354] alg: No test for adler32 (adler32-zlib) [10562.926674] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [10563.129942] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [10564.792290] Key type lgssc registered [10566.217112] Lustre: Echo OBD driver; http://www.lustre.org/ [10574.832153] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10574.843991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10576.393043] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10582.514909] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10586.598394] LustreError: 329554:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10586.619184] LustreError: 329554:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [10591.715388] LustreError: 329553:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10593.168488] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10593.739099] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10599.380377] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10609.395550] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10617.238338] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10624.485177] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10624.485462] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10624.486909] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10624.508021] Lustre: Skipped 1 previous similar message [10626.736597] Lustre: server umount lustre-MDT0000 complete [10629.608186] LustreError: 330282:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10629.633088] LustreError: 330282:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10631.588984] LustreError: 329534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980597 with bad export cookie 11582033966061255093 [10631.596104] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10631.607749] LustreError: 329534:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10631.961904] Lustre: server umount lustre-MDT0001 complete [10648.949689] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10649.346637] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [10649.422757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10649.725248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10654.987504] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10665.646577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10665.802380] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [10665.888186] Lustre: 332143:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10666.141528] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10671.612929] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10680.785648] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10688.137074] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10700.077065] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10700.305840] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [10700.321820] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [10700.337204] LustreError: 334027:0:(tgt_mount.c:2613:server_fill_super()) Unable to start targets: -2 [10700.343285] LustreError: 334027:0:(tgt_mount.c:2128:server_put_super()) no obd lustre-OST0000 [10700.347422] LustreError: 334027:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [10700.425220] Lustre: server umount lustre-OST0000 complete [10700.428385] LustreError: 334027:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [10711.793271] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10711.984892] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10712.287486] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10719.690979] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10722.477856] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [10729.742400] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10739.011109] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10739.180679] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [10739.244215] Lustre: lustre-OST0001: new disk, initializing [10739.247840] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [10739.261936] Lustre: 335570:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10739.355497] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [10746.704739] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10749.007439] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [10749.014338] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [10749.217504] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [10756.979206] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10764.781235] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10764.783070] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10764.795573] LustreError: Skipped 1 previous similar message [10764.825458] Lustre: Skipped 1 previous similar message [10764.842649] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [10764.856161] Lustre: Skipped 2 previous similar messages [10767.351996] Lustre: server umount lustre-OST0001 complete [10769.910853] LustreError: 336374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10769.925634] LustreError: 336374:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10773.473486] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10773.491434] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10773.511906] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10775.023946] LustreError: 336374:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10775.064227] LustreError: 336374:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10777.406119] Lustre: server umount lustre-OST0000 complete [10784.225900] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10784.240917] LustreError: Skipped 1 previous similar message [10784.247724] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10784.255438] Lustre: Skipped 1 previous similar message [10784.265376] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10784.275464] Lustre: Skipped 3 previous similar messages [10788.407762] Lustre: server umount lustre-MDT0000 complete [10790.380581] LustreError: 332153:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10790.405324] LustreError: 332153:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [10793.246444] LustreError: 335569:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788980758 with bad export cookie 11582033966061255646 [10793.264442] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10793.703848] Lustre: server umount lustre-MDT0001 complete [10806.378101] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [10809.522634] Key type lgssc unregistered [10809.807390] LNet: 337527:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10809.812341] LNetError: 337527:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10809.828566] LNet: Removed LNI 192.168.201.106@tcp [10810.822092] Key type .llcrypt unregistered [10810.823348] Key type ._llcrypt unregistered [10889.900513] Key type ._llcrypt registered [10889.903567] Key type .llcrypt registered [10890.055396] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [10908.960313] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [10910.599629] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10910.746779] alg: No test for adler32 (adler32-zlib) [10911.828230] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [10912.080485] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [10913.808224] Key type lgssc registered [10915.099055] Lustre: Echo OBD driver; http://www.lustre.org/ [10960.045052] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10960.093146] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10961.449802] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10961.518734] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10961.599662] Lustre: lustre-MDT0000: new disk, initializing [10961.692409] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10961.712817] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10966.743350] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10979.271512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10979.465605] Lustre: 342465:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10979.520724] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10979.525915] Lustre: Skipped 1 previous similar message [10979.624869] Lustre: lustre-MDT0001: new disk, initializing [10979.730182] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10979.795605] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10979.811208] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10985.793534] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [10998.322951] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11006.105878] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11014.635246] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11014.937926] Lustre: lustre-OST0000: new disk, initializing [11014.944624] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11014.953562] Lustre: 344433:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11015.066494] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11022.900648] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11022.906165] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11023.037885] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11023.304992] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11034.298146] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11039.293872] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [11039.481996] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [11045.136309] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [11045.403813] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [11048.929977] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11048.936021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11048.941995] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11048.964323] Lustre: Skipped 1 previous similar message [11048.999472] Lustre: Skipped 1 previous similar message [11053.986691] Lustre: server umount lustre-OST0000 complete [11059.172147] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11059.200282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11059.209104] Lustre: Skipped 1 previous similar message [11064.290516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11064.298340] Lustre: Skipped 1 previous similar message [11064.832429] Lustre: server umount lustre-MDT0000 complete [11069.026255] LustreError: 342454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981034 with bad export cookie 4450325947426022899 [11069.034637] LustreError: 342454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11069.038143] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11069.362661] Lustre: server umount lustre-MDT0001 complete [11080.545395] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 15:10:43 (1788981043) [11090.480770] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 15:10:53 (1788981053) [11101.888665] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11102.432594] LustreError: 347190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11102.483802] LustreError: 347190:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11102.595771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11107.815112] LustreError: 347191:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11108.803515] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11112.934641] LustreError: 347190:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11118.051219] LustreError: 347191:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11120.673066] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11121.204251] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11126.399609] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11136.234779] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11143.409898] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11151.459603] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11151.768333] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11159.435450] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11168.554910] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11182.714955] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [11185.054776] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [11187.662655] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [11189.919730] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [11192.054785] Lustre: Failing over lustre-MDT0000 [11192.510437] Lustre: server umount lustre-MDT0000 complete [11192.810974] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11192.824046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11192.845960] Lustre: Skipped 1 previous similar message [11192.851968] LustreError: 347191:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11194.858296] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11194.885195] Lustre: Skipped 1 previous similar message [11205.102501] LustreError: 347191:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11205.132752] LustreError: 347191:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 11 previous similar messages [11211.233814] Lustre: 340684:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788981160/real 1788981160] req@0000000094b67463 x1875882412158592/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788981176 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11211.258585] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11214.141076] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11220.465488] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x3dc2bbdecf082d78 [11220.477768] Lustre: MGC192.168.201.106@tcp: Connection restored to 0@lo (at 0@lo) [11221.011865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11221.158533] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11221.497827] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11226.119076] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11226.147253] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [11226.225863] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [11228.204404] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11240.932311] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11243.112097] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11248.365867] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [11251.561041] Lustre: Failing over lustre-MDT0000 [11251.681606] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11251.688877] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11251.707523] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11251.710757] Lustre: Skipped 2 previous similar messages [11251.728791] Lustre: Skipped 2 previous similar messages [11252.121189] Lustre: server umount lustre-MDT0000 complete [11252.267849] LustreError: 349681:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [11252.298440] LustreError: 349681:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 15 previous similar messages [11273.187745] Lustre: 340685:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788981222/real 1788981222] req@00000000bd37037f x1875882412182272/t0(0) o400->MGC192.168.201.106@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788981238 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11273.237894] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11273.967143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11282.401059] LustreError: 340682:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000001b39b5e4 x1875882412187392/t0(0) o250->MGC192.168.201.106@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [11282.808262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11282.918844] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [11282.958269] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [11288.042287] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [11288.047737] Lustre: Skipped 2 previous similar messages [11288.071939] Lustre: lustre-MDT0000: Recovery over after 0:06, of 2 clients 2 recovered and 0 were evicted. [11288.129744] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [11288.678254] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11300.686785] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11303.122495] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [11313.635847] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11313.659859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11313.664703] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11313.665325] Lustre: Skipped 1 previous similar message [11317.305844] Lustre: server umount lustre-MDT0000 complete [11318.755520] LustreError: 349683:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11318.773691] LustreError: 349683:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 29 previous similar messages [11321.509712] LustreError: 347170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981287 with bad export cookie 4450325947426025615 [11321.512507] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11321.527600] LustreError: 347170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11321.811591] Lustre: server umount lustre-MDT0001 complete [11336.826715] Lustre: server umount lustre-OST0000 complete [11355.535064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11356.063549] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11361.865808] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11373.187375] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11379.924619] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11389.993516] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11398.302687] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11409.308280] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11417.250219] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11425.342218] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [11427.306690] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11435.499617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11435.513880] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11435.518012] LustreError: Skipped 1 previous similar message [11435.518879] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11435.518883] Lustre: Skipped 1 previous similar message [11435.572445] Lustre: Skipped 3 previous similar messages [11440.090225] Lustre: server umount lustre-OST0000 complete [11445.225156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11450.853472] LustreError: 353558:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11450.874592] LustreError: 353558:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [11451.156370] Lustre: server umount lustre-MDT0000 complete [11455.626624] LustreError: 356340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981421 with bad export cookie 4450325947426026357 [11455.627291] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11455.652528] LustreError: 356340:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11456.217605] Lustre: server umount lustre-MDT0001 complete [11471.360715] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [11474.450617] Key type lgssc unregistered [11474.718776] LNet: 357091:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11474.728891] LNetError: 357091:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11474.753865] LNet: Removed LNI 192.168.201.106@tcp [11475.870897] Key type .llcrypt unregistered [11475.872537] Key type ._llcrypt unregistered [11508.221665] Key type ._llcrypt registered [11508.229231] Key type .llcrypt registered [11508.386191] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11509.944931] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11510.226617] alg: No test for adler32 (adler32-zlib) [11511.245901] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [11511.436613] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [11513.093078] Key type lgssc registered [11514.112988] Lustre: Echo OBD driver; http://www.lustre.org/ [11521.845386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11521.877923] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11523.345082] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11527.896202] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11533.794505] LustreError: 358605:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11533.821365] LustreError: 358605:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11536.520356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11536.899419] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11542.320955] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11551.438213] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11557.726621] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11565.711649] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11566.288626] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11568.375958] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11573.261723] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11581.647094] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11597.298462] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11597.300258] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11597.326737] Lustre: Skipped 1 previous similar message [11597.339479] Lustre: Skipped 2 previous similar messages [11601.820030] Lustre: server umount lustre-MDT0000 complete [11602.405393] LustreError: 358602:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11602.428890] LustreError: 358602:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [11606.493693] LustreError: 358588:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981572 with bad export cookie 12066775530695975181 [11606.522632] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11606.969330] Lustre: server umount lustre-MDT0001 complete [11622.540644] Lustre: server umount lustre-OST0000 complete [11640.304041] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11640.946603] LustreError: 362338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11641.048252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11646.293553] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11646.435798] LustreError: 362339:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11646.459994] LustreError: 362339:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [11650.533718] LustreError: 362338:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11657.337395] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11657.886400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11663.260522] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11673.236725] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11680.021124] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11688.270038] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11688.624900] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11696.251639] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11706.215341] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11711.160419] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [11726.822131] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11726.834249] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11726.836885] Lustre: Skipped 3 previous similar messages [11726.848150] Lustre: Skipped 2 previous similar messages [11731.314442] Lustre: server umount lustre-MDT0000 complete [11731.939223] LustreError: 364640:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11731.949210] LustreError: 364640:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [11735.226261] LustreError: 362318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981700 with bad export cookie 12066775530695976294 [11735.237135] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11735.251022] LustreError: 362318:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [11735.621290] Lustre: server umount lustre-MDT0001 complete [11750.511531] Lustre: server umount lustre-OST0000 complete [11767.948632] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11768.539878] LustreError: 366254:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11768.674734] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11774.206497] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11784.089772] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11789.444920] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11797.466178] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11804.072280] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11811.789534] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11812.036447] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11812.041510] Lustre: Skipped 1 previous similar message [11814.131507] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [11818.891159] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [11826.724481] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11834.863922] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [11834.870278] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11834.879411] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11834.907028] Lustre: Skipped 1 previous similar message [11839.393457] Lustre: server umount lustre-OST0000 complete [11845.092789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11845.125267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11845.140132] Lustre: Skipped 2 previous similar messages [11849.676396] Lustre: server umount lustre-MDT0000 complete [11850.210816] LustreError: 368830:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11850.237532] LustreError: 368830:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [11853.407452] LustreError: 366234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788981819 with bad export cookie 12066775530695977316 [11853.415938] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11853.418437] LustreError: 366234:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [11853.725946] Lustre: server umount lustre-MDT0001 complete [11866.284374] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [11869.242781] Key type lgssc unregistered [11869.657744] LNet: 369782:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11869.672475] LNetError: 369782:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11870.702782] LNet: Removed LNI 192.168.201.106@tcp [11871.693820] Key type .llcrypt unregistered [11871.695960] Key type ._llcrypt unregistered [11900.177692] Key type ._llcrypt registered [11900.179792] Key type .llcrypt registered [11900.263594] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11901.810350] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11901.893681] alg: No test for adler32 (adler32-zlib) [11903.076875] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [11903.433844] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [11905.203085] Key type lgssc registered [11906.690181] Lustre: Echo OBD driver; http://www.lustre.org/ [11916.554393] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11916.586922] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11916.596910] LustreError: 371223:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [11916.621501] LustreError: 371223:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [11916.633909] LustreError: 371223:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [11916.642585] LustreError: 371223:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [11916.650245] LustreError: 371223:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11925.741603] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [11925.749352] LustreError: 371620:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [11925.777963] LustreError: 371620:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [11925.788881] LustreError: 371620:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [11925.795948] LustreError: 371620:0:(tgt_mount.c:2581:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [11925.802420] LustreError: 371620:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [11950.033017] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [11953.313989] Key type lgssc unregistered [11953.631668] LNet: 372623:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11953.641980] LNetError: 372623:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11953.661617] LNet: Removed LNI 192.168.201.106@tcp [11954.846294] Key type .llcrypt unregistered [11954.851447] Key type ._llcrypt unregistered [11975.202809] Key type ._llcrypt registered [11975.208776] Key type .llcrypt registered [11975.376426] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [11997.231222] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [11999.303220] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11999.571192] alg: No test for adler32 (adler32-zlib) [12000.996072] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [12001.473629] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [12003.160521] Key type lgssc registered [12003.897266] Lustre: Echo OBD driver; http://www.lustre.org/ [12048.316399] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12048.363354] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12049.750717] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12049.815420] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12049.934116] Lustre: lustre-MDT0000: new disk, initializing [12050.068096] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12050.101399] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12055.670538] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12067.841933] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12068.057812] Lustre: 375781:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12068.150860] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12068.163109] Lustre: Skipped 1 previous similar message [12068.309714] Lustre: lustre-MDT0001: new disk, initializing [12068.511791] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12068.561554] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12068.582198] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12073.226091] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12085.211676] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12092.060182] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12100.055911] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12100.319482] Lustre: lustre-OST0000: new disk, initializing [12100.323778] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12100.331504] Lustre: 377750:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12100.476706] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12105.859387] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12105.866053] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12106.045446] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12107.349346] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12118.529530] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12124.247937] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12124.548118] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12129.590375] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12129.867485] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12131.812867] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12131.822270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12131.830374] LustreError: Skipped 1 previous similar message [12131.842555] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12136.929816] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12136.944456] Lustre: Skipped 2 previous similar messages [12142.057286] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12142.069242] Lustre: Skipped 1 previous similar message [12146.144572] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12146.310570] Lustre: server umount lustre-OST0000 complete [12150.245664] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12150.255753] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12150.267550] Lustre: Skipped 1 previous similar message [12150.273352] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12152.302257] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12156.222282] Lustre: server umount lustre-MDT0000 complete [12160.136302] LustreError: 375771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788982125 with bad export cookie 14644672487455208046 [12160.143030] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12160.149187] LustreError: 375771:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12160.810876] Lustre: server umount lustre-MDT0001 complete [12170.816310] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [12173.009729] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 15:28:56 (1788982136) [12180.968739] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12181.512067] LustreError: 380209:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12181.544324] LustreError: 380209:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12181.648689] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12187.106124] LustreError: 380210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12187.599863] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12192.233560] LustreError: 380209:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12197.352328] LustreError: 380210:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12198.858659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12199.260735] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12204.315430] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12214.683498] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12222.805572] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12230.574030] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12230.908039] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12238.383822] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12247.936460] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12256.308135] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12256.527647] Lustre: lustre-OST0001: new disk, initializing [12256.531872] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [12256.539280] Lustre: 383049:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12256.677406] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [12262.457308] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [12262.471370] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [12262.641527] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [12262.877944] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12274.131889] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12281.382882] Lustre: lustre-OST0001: Not available for connect from 192.168.201.6@tcp (stopping) [12281.385443] Lustre: Skipped 2 previous similar messages [12283.361899] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12283.384551] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12283.408126] Lustre: Skipped 1 previous similar message [12284.593977] Lustre: server umount lustre-OST0001 complete [12286.475729] LustreError: 383373:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [12298.724047] LustreError: 382035:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12298.768756] LustreError: 382035:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [12303.865801] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12303.870132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12303.885032] Lustre: Skipped 1 previous similar message [12303.902295] Lustre: Skipped 2 previous similar messages [12309.999887] Lustre: server umount lustre-OST0000 complete [12317.154995] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12317.166234] LustreError: Skipped 1 previous similar message [12317.172603] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12322.164464] Lustre: server umount lustre-MDT0000 complete [12324.322668] LustreError: 380209:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12324.353719] LustreError: 380209:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [12328.618071] LustreError: 380191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788982294 with bad export cookie 14644672487455208928 [12328.628915] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12328.647371] LustreError: 380191:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [12335.557966] Lustre: server umount lustre-MDT0001 complete [12349.089516] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12352.665727] Key type lgssc unregistered [12353.045379] LNet: 385036:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12353.052690] LNetError: 385036:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12353.081988] LNet: Removed LNI 192.168.201.106@tcp [12354.247108] Key type .llcrypt unregistered [12354.252218] Key type ._llcrypt unregistered [12416.246699] Key type ._llcrypt registered [12416.251957] Key type .llcrypt registered [12416.407428] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12418.153845] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12418.215167] alg: No test for adler32 (adler32-zlib) [12419.422046] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [12419.719909] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [12421.440300] Key type lgssc registered [12422.625408] Lustre: Echo OBD driver; http://www.lustre.org/ [12431.727545] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12431.780830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12433.100622] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [12433.135220] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12433.323354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12438.187242] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12448.831568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12449.021490] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [12449.050326] Lustre: 387837:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12449.565631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12455.724831] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12464.628404] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12471.932249] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12475.361134] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12475.365576] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12475.378165] Lustre: Skipped 1 previous similar message [12475.399230] Lustre: Skipped 1 previous similar message [12480.483359] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12480.490554] Lustre: Skipped 1 previous similar message [12481.434744] Lustre: server umount lustre-MDT0000 complete [12485.601125] LustreError: 387849:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12485.628146] LustreError: 387849:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12485.849440] LustreError: 387829:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788982451 with bad export cookie 1013487956317779675 [12485.873204] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12486.419115] Lustre: server umount lustre-MDT0001 complete [12498.209597] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 15:34:21 (1788982461) [12506.283132] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12506.597366] LustreError: 390348:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12506.662297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12511.713401] LustreError: 390349:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12512.128960] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12516.833471] LustreError: 390367:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12521.656290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12522.072215] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12526.869443] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12536.558922] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12543.573533] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12554.121877] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12554.520232] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [12554.905697] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12563.440962] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12572.744523] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12581.550262] Lustre: Permanently deactivating lustre-OST0000 [12581.575296] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [12591.157941] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12593.182274] LustreError: 393388:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [12593.197438] LustreError: 393388:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12593.212384] LustreError: 393388:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12597.285614] LustreError: 393437:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12597.296864] LustreError: 393437:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12602.308535] Lustre: server umount lustre-OST0000 complete [12605.566559] LustreError: 393686:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [12605.571915] LustreError: 393686:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [12608.997446] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12609.012142] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12609.042988] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12610.532078] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12615.655490] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12615.667122] Lustre: Skipped 3 previous similar messages [12620.770191] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12620.793718] Lustre: Skipped 1 previous similar message [12622.304249] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [12622.500063] Lustre: server umount lustre-MDT0000 complete [12625.889582] LustreError: 390344:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12625.928417] LustreError: 390344:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12627.499121] LustreError: 390330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788982593 with bad export cookie 1013487956317780256 [12627.508150] LustreError: 390330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12627.511201] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12638.323439] Lustre: server umount lustre-MDT0001 complete [12647.654276] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12652.365317] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12655.117745] LustreError: 394846:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [12661.530185] Lustre: 395185:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.201.106@tcp [12663.782470] Lustre: server umount MGS complete [12677.527556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12678.612759] LustreError: 395910:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12678.641641] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12678.660268] Lustre: Skipped 1 previous similar message [12678.774411] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12685.017663] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12699.111365] LustreError: 395911:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12699.143681] LustreError: 395911:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [12700.531152] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12701.098469] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [12701.191514] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12706.969122] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12715.926153] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12722.832669] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12725.248955] Lustre: Permanently reactivating lustre-OST0000 [12725.290461] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [12725.296586] Lustre: Skipped 2 previous similar messages [12753.890735] LustreError: 397693:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [12763.153981] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12763.541132] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12770.824376] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12780.406437] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12789.217946] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12789.220035] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12789.234842] LustreError: Skipped 1 previous similar message [12789.247967] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12789.263756] Lustre: Skipped 1 previous similar message [12792.391556] Lustre: server umount lustre-OST0000 complete [12798.437076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12798.444124] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12802.809910] Lustre: server umount lustre-MDT0000 complete [12804.582233] LustreError: 395911:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12804.605725] LustreError: 395911:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [12806.770728] LustreError: 395892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788982772 with bad export cookie 1013487956317781306 [12806.774985] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12806.784225] LustreError: 395892:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12807.173427] Lustre: server umount lustre-MDT0001 complete [12817.950105] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [12821.660395] Key type lgssc unregistered [12822.058919] LNet: 400276:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12822.066663] LNetError: 400276:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12822.080233] LNet: Removed LNI 192.168.201.106@tcp [12823.078405] Key type .llcrypt unregistered [12823.080082] Key type ._llcrypt unregistered [12844.698877] Key type ._llcrypt registered [12844.701749] Key type .llcrypt registered [12844.819682] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [12862.486854] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [12863.852953] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12863.924495] alg: No test for adler32 (adler32-zlib) [12865.172225] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [12865.498180] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [12867.200301] Key type lgssc registered [12868.656370] Lustre: Echo OBD driver; http://www.lustre.org/ [12914.698847] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12914.719417] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12916.122049] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12916.186928] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12916.309395] Lustre: lustre-MDT0000: new disk, initializing [12916.448462] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12916.536089] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12922.604491] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12936.612063] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12936.763532] Lustre: 403436:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12936.809275] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12936.813321] Lustre: Skipped 1 previous similar message [12936.901143] Lustre: lustre-MDT0001: new disk, initializing [12937.011789] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12937.055913] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12937.069033] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12942.525966] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12951.678772] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12958.385288] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12967.106322] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12967.495921] Lustre: lustre-OST0000: new disk, initializing [12967.503908] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12967.510126] Lustre: 405405:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12967.626306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12973.618183] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12973.639806] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12973.773284] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12974.568571] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [12984.699339] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12989.785219] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12990.010337] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12994.656653] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12994.860378] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12999.660571] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12999.670376] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12999.670912] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12999.689667] Lustre: Skipped 1 previous similar message [13003.064630] Lustre: server umount lustre-OST0000 complete [13008.871382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13008.881306] LustreError: Skipped 1 previous similar message [13008.891718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13008.911155] Lustre: Skipped 1 previous similar message [13008.923630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13009.898253] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13013.853746] Lustre: server umount lustre-MDT0000 complete [13018.363144] LustreError: 403427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788982984 with bad export cookie 16852523858469650916 [13018.377275] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13018.380575] LustreError: 403427:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13018.828654] Lustre: server umount lustre-MDT0001 complete [13029.840778] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [13032.095232] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 15:43:15 (1788982995) [13040.820723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13041.378328] LustreError: 407867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13041.392475] LustreError: 407867:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13041.496725] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13046.763174] LustreError: 407868:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13046.943558] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13051.875220] LustreError: 407867:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13056.601105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13057.181304] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13062.609290] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13072.326887] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13079.620039] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13087.777949] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13088.159760] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13095.340121] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13104.756864] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13107.058659] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [13114.263226] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [13127.139977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13127.141935] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13127.164905] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13127.172328] Lustre: Skipped 2 previous similar messages [13128.443871] Lustre: server umount lustre-OST0000 complete [13133.796543] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13133.803316] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13133.824316] Lustre: Skipped 1 previous similar message [13133.834199] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13133.844430] Lustre: Skipped 1 previous similar message [13139.397632] Lustre: server umount lustre-MDT0000 complete [13142.507723] LustreError: 409655:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13142.523121] LustreError: 409655:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13143.955705] LustreError: 408602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788983109 with bad export cookie 16852523858469651798 [13143.961970] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13143.964368] LustreError: 408602:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13144.517632] Lustre: server umount lustre-MDT0001 complete [13157.660807] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13160.810313] Key type lgssc unregistered [13161.237227] LNet: 411595:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13161.254580] LNetError: 411595:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13161.291128] LNet: Removed LNI 192.168.201.106@tcp [13162.396253] Key type .llcrypt unregistered [13162.398843] Key type ._llcrypt unregistered [13187.848889] Key type ._llcrypt registered [13187.852065] Key type .llcrypt registered [13187.977171] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13188.296723] Key type .llcrypt unregistered [13188.299446] Key type ._llcrypt unregistered [13209.908897] Key type ._llcrypt registered [13209.911845] Key type .llcrypt registered [13210.017719] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13210.896627] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13211.028694] alg: No test for adler32 (adler32-zlib) [13212.389686] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [13212.623706] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [13214.344246] Key type lgssc registered [13215.793314] Lustre: Echo OBD driver; http://www.lustre.org/ [13223.854251] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13223.875330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13225.694809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13231.259723] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13236.196990] LustreError: 413804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13236.219219] LustreError: 413804:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13241.350509] LustreError: 413805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13242.464717] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13242.892369] LustreError: 413804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13242.933593] LustreError: 413804:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13242.960310] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13247.975025] LustreError: 413801:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13248.001198] LustreError: 413801:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13250.788481] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13251.073071] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [13256.172050] LustreError: 413804:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13256.209622] LustreError: 413804:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13259.473795] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13266.409408] LustreError: 413805:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13266.431194] LustreError: 413805:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 3 previous similar messages [13267.270198] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13267.864538] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13273.226598] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13286.884816] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13286.894801] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13286.903459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13288.423498] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13288.432457] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13289.934350] Lustre: server umount lustre-OST0000 complete [13297.610131] Lustre: server umount lustre-MDT0000 complete [13298.660438] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13298.692658] LustreError: 415543:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13298.713852] LustreError: 415543:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13302.354640] LustreError: 413786:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788983268 with bad export cookie 2816993183356486960 [13302.363833] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13303.125066] Lustre: server umount lustre-MDT0001 complete [13317.251868] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13320.181349] Key type lgssc unregistered [13320.481358] LNet: 416955:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13320.494471] LNetError: 416955:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13320.514987] LNet: Removed LNI 192.168.201.106@tcp [13321.566112] Key type .llcrypt unregistered [13321.570248] Key type ._llcrypt unregistered [13351.173936] Key type ._llcrypt registered [13351.175988] Key type .llcrypt registered [13351.310679] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13352.536809] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13352.562059] alg: No test for adler32 (adler32-zlib) [13353.949352] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [13354.190355] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [13355.888596] Key type lgssc registered [13357.125033] Lustre: Echo OBD driver; http://www.lustre.org/ [13367.338862] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13367.385972] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13382.944400] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [13388.064355] LustreError: 418343:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.201.106@tcp: failed processing log, type 4: rc = -110 [13414.880635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13423.396600] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13432.962848] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13442.224551] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13442.926901] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13443.009142] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [13448.166860] LustreError: 418369:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13448.189682] LustreError: 418369:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13448.865398] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13450.215822] LustreError: 419380:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13450.254243] LustreError: 419380:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13455.333485] LustreError: 419713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13455.369479] LustreError: 419713:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13459.345625] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13459.664343] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13465.624216] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13475.366818] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13482.325989] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13490.670056] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13490.674424] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13490.692712] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13490.710086] Lustre: Skipped 1 previous similar message [13491.629559] Lustre: server umount lustre-OST0000 complete [13496.245348] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [13496.262293] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13496.278772] Lustre: Skipped 1 previous similar message [13496.291215] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13500.908315] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13500.909110] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13502.553140] Lustre: server umount lustre-MDT0000 complete [13506.024557] LustreError: 420873:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13506.048665] LustreError: 420873:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13506.289143] LustreError: 418350:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788983471 with bad export cookie 5395682995114708715 [13506.301399] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13506.651307] Lustre: server umount lustre-MDT0001 complete [13519.187212] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13522.219329] Key type lgssc unregistered [13522.478655] LNet: 421915:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13522.486714] LNetError: 421915:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13522.502866] LNet: Removed LNI 192.168.201.106@tcp [13523.552600] Key type .llcrypt unregistered [13523.556594] Key type ._llcrypt unregistered [13550.694994] Key type ._llcrypt registered [13550.697585] Key type .llcrypt registered [13550.783533] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13551.801762] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13551.983042] alg: No test for adler32 (adler32-zlib) [13553.362581] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [13553.708840] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [13555.384763] Key type lgssc registered [13556.579389] Lustre: Echo OBD driver; http://www.lustre.org/ [13565.499804] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13565.534557] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13566.964242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13571.854447] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13577.186118] LustreError: 423331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13577.209784] LustreError: 423331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13581.928846] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13582.357462] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13588.244364] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13596.865022] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13603.531046] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13611.500570] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13611.864981] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13612.926441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [13618.241083] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13625.996783] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13632.484226] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13632.486987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13632.513024] Lustre: Skipped 1 previous similar message [13632.526538] Lustre: Skipped 2 previous similar messages [13634.080557] Lustre: lustre-MDT0000: Not available for connect from 192.168.201.6@tcp (stopping) [13637.614598] LustreError: 425170:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13637.822851] Lustre: server umount lustre-MDT0000 complete [13639.162433] LustreError: 423327:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [13639.184028] LustreError: 423327:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13640.211586] LustreError: 423310:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.201.6@tcp arrived at 1788983605 with bad export cookie 3213726833023807085 [13642.437808] LustreError: 423312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788983608 with bad export cookie 3213726833023806189 [13642.448453] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13642.451858] LustreError: 423312:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13642.726667] LustreError: 423331:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13642.745091] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13642.753040] LustreError: 423331:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13642.757766] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13642.785298] Lustre: Skipped 1 previous similar message [13642.795065] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13647.849078] LustreError: 423330:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13647.849540] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13647.867482] LustreError: 423330:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13648.662569] Lustre: server umount lustre-OST0000 complete [13656.982656] Lustre: server umount lustre-MDT0001 complete [13669.372610] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13672.330284] Key type lgssc unregistered [13672.576206] LNet: 427014:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13672.587198] LNetError: 427014:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13672.612459] LNet: Removed LNI 192.168.201.106@tcp [13673.456362] Key type .llcrypt unregistered [13673.458917] Key type ._llcrypt unregistered [13700.366248] Key type ._llcrypt registered [13700.369199] Key type .llcrypt registered [13700.502931] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13702.033488] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13702.083510] alg: No test for adler32 (adler32-zlib) [13703.102958] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [13703.416835] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [13705.066257] Key type lgssc registered [13706.355601] Lustre: Echo OBD driver; http://www.lustre.org/ [13714.546325] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13714.572141] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13716.244237] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13721.752962] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13725.670388] LustreError: 428428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13725.699044] LustreError: 428428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13730.791175] LustreError: 428429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13732.527311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13732.828980] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13737.799241] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13745.910890] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13752.911127] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13761.815378] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13762.337293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13771.011783] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13780.488705] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13785.949282] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [13791.203623] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13791.227745] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [13791.227867] Lustre: Skipped 1 previous similar message [13791.254100] Lustre: Skipped 1 previous similar message [13791.770350] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [13795.465683] Lustre: server umount lustre-MDT0001 complete [13796.336264] LustreError: 428429:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13796.355828] LustreError: 428429:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [13801.453693] LustreError: 428428:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13801.485477] LustreError: 428428:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [13806.577957] LustreError: 429156:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13806.582166] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13806.608058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13806.632876] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13808.127448] Lustre: server umount lustre-OST0000 complete [13813.995730] Lustre: server umount lustre-MDT0000 complete [13831.339408] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [13835.194498] Key type lgssc unregistered [13835.499874] LNet: 432115:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13835.508371] LNetError: 432115:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13835.528816] LNet: Removed LNI 192.168.201.106@tcp [13836.507477] Key type .llcrypt unregistered [13836.510134] Key type ._llcrypt unregistered [13866.018364] Key type ._llcrypt registered [13866.021747] Key type .llcrypt registered [13866.167945] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [13885.228725] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13886.383703] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13886.499080] alg: No test for adler32 (adler32-zlib) [13887.750822] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [13888.073286] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [13889.826487] Key type lgssc registered [13891.064040] Lustre: Echo OBD driver; http://www.lustre.org/ [13933.351804] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [13950.838284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13950.879290] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13952.192456] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13952.210097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13952.268495] Lustre: lustre-MDT0000: new disk, initializing [13952.318613] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13952.337083] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13957.514213] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13973.762872] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13974.066929] Lustre: lustre-OST0000: new disk, initializing [13974.070829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13974.074329] Lustre: Skipped 1 previous similar message [13974.080386] Lustre: 437150:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13974.174610] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13980.553620] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [13982.815950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13982.828853] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13982.864107] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13991.729986] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14008.983876] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14009.124975] Lustre: 436132:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14009.219193] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14009.302790] Lustre: lustre-MDT0001: new disk, initializing [14009.433127] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14009.470518] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [14009.482804] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [14014.861201] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14027.021849] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [14027.310148] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [14032.709322] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [14032.920283] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [14039.013942] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14039.041129] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [14039.042317] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [14039.053384] Lustre: Skipped 1 previous similar message [14039.602431] Lustre: lustre-MDT0001: Not available for connect from 192.168.201.6@tcp (stopping) [14043.886897] Lustre: server umount lustre-MDT0001 complete [14049.254573] LustreError: 436144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14049.308265] LustreError: 436144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 5 previous similar messages [14049.793111] LustreError: 436140:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 192.168.201.6@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [14054.377879] LustreError: 436143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14054.404357] LustreError: 436143:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14059.492864] LustreError: 437157:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14059.518202] LustreError: 437157:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14064.613186] LustreError: 436143:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14064.614080] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14064.652838] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14064.666199] Lustre: Skipped 1 previous similar message [14064.682946] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14069.729638] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14070.787332] Lustre: server umount lustre-OST0000 complete [14074.850263] LustreError: 436144:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14074.871708] LustreError: 436144:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14082.036191] Lustre: server umount lustre-MDT0000 complete [14099.222178] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14102.056865] Key type lgssc unregistered [14102.358791] LNet: 440909:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14102.370920] LNetError: 440909:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14102.387265] LNet: Removed LNI 192.168.201.106@tcp [14103.507793] Key type .llcrypt unregistered [14103.511799] Key type ._llcrypt unregistered [14153.774370] Key type ._llcrypt registered [14153.775901] Key type .llcrypt registered [14153.974552] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [14154.391876] Key type .llcrypt unregistered [14154.393407] Key type ._llcrypt unregistered [14172.283556] Key type ._llcrypt registered [14172.285365] Key type .llcrypt registered [14172.405152] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [14173.269708] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [14173.306898] alg: No test for adler32 (adler32-zlib) [14174.556636] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [14174.821964] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [14176.521827] Key type lgssc registered [14177.887126] Lustre: Echo OBD driver; http://www.lustre.org/ [14208.825666] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [14208.855765] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14210.149104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [14210.184491] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [14210.313657] Lustre: lustre-MDT0000: new disk, initializing [14210.478293] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14210.521794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [14216.552529] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14229.936553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14230.168982] Lustre: 444279:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [14230.260829] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [14230.273591] Lustre: Skipped 1 previous similar message [14230.418425] Lustre: lustre-MDT0001: new disk, initializing [14230.622543] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14230.728967] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [14230.749035] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [14236.284198] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14247.976786] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14255.418499] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14263.767676] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14264.111290] Lustre: lustre-OST0000: new disk, initializing [14264.123417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [14264.131892] Lustre: 446248:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [14264.241714] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14270.606416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [14270.629088] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [14270.818337] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [14271.839470] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14283.175875] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14291.432198] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14291.435914] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14291.446828] Lustre: Skipped 1 previous similar message [14291.465687] Lustre: Skipped 2 previous similar messages [14296.546633] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14296.570557] Lustre: Skipped 2 previous similar messages [14297.246781] Lustre: server umount lustre-MDT0000 complete [14302.169206] LustreError: 444272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788984267 with bad export cookie 14528143886590715266 [14302.175913] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14302.192361] LustreError: 444272:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14303.049044] Lustre: server umount lustre-MDT0001 complete [14319.862499] Lustre: server umount lustre-OST0000 complete [14332.051991] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:04:55 (1788984295) [14341.675541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14342.099708] LustreError: 448277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14342.119168] LustreError: 448277:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [14342.192885] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [14347.245262] LustreError: 448278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14347.585853] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14351.329859] LustreError: 448277:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14356.456953] LustreError: 448278:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14357.832965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [14358.436792] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [14364.756848] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14374.201430] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [14380.261532] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [14401.272235] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [14401.573542] Lustre: Found index 0 for lustre-OST0000, updating log [14401.963067] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [14409.052399] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [14413.244902] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [14418.072595] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [14423.542620] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14423.545446] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [14423.553568] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [14423.570853] Lustre: Skipped 2 previous similar messages [14429.257866] Lustre: server umount lustre-OST0000 complete [14445.708377] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14445.717537] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14445.727103] Lustre: Skipped 3 previous similar messages [14450.679294] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [14450.688840] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [14450.715645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [14459.873642] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [14460.006402] Lustre: server umount lustre-MDT0000 complete [14463.458573] LustreError: 449713:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [14463.475878] LustreError: 449713:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [14463.584387] LNet: 1 local NIs in recovery (showing 1): 192.168.201.106@tcp [14464.090830] LustreError: 448258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788984429 with bad export cookie 14528143886590716659 [14464.096512] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [14464.099439] LustreError: 448258:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [14464.577915] Lustre: server umount lustre-MDT0001 complete [14477.528383] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:07:20 (1788984440) [15119.376155] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15119.993939] LustreError: 454695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15120.113582] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15125.498042] LustreError: 454695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15125.521851] LustreError: 454695:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15125.902882] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15129.634854] LustreError: 454696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15135.720270] LustreError: 454696:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15135.738457] LustreError: 454696:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15136.867855] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15137.365763] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15143.155940] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15152.973657] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15161.618856] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15180.449883] LNet: 1 local NIs in recovery (showing 1): 192.168.201.106@tcp [15185.470604] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15185.867241] Lustre: Found index 0 for lustre-OST0000, updating log [15186.102264] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15193.868243] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15194.282441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [15204.044448] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15209.970210] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [15209.971295] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15210.025578] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15210.030482] Lustre: Skipped 8 previous similar messages [15221.217064] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [15221.427090] Lustre: server umount lustre-OST0000 complete [15227.109752] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15227.117073] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15227.143480] Lustre: Skipped 1 previous similar message [15227.155054] Lustre: Skipped 1 previous similar message [15229.218876] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15229.241378] Lustre: Skipped 3 previous similar messages [15232.305074] LustreError: 454695:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [15232.331704] LustreError: 454695:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [15232.501855] Lustre: server umount lustre-MDT0000 complete [15237.195554] LustreError: 454675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788985202 with bad export cookie 14528143886590717737 [15237.201751] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15237.209457] LustreError: 454675:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [15237.502607] Lustre: server umount lustre-MDT0001 complete [15240.864727] LNet: 1 local NIs in recovery (showing 1): 192.168.201.106@tcp [15272.722898] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [15286.395673] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [15337.034282] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:21:39 (1788985299) [15362.760585] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [15366.220946] Key type lgssc unregistered [15366.516204] LNet: 461312:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15366.524905] LNetError: 461312:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15447.464783] LNet: Removed LNI 192.168.201.106@tcp [15448.836122] Key type .llcrypt unregistered [15448.840297] Key type ._llcrypt unregistered [15470.577671] Key type ._llcrypt registered [15470.579737] Key type .llcrypt registered [15470.763923] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [15472.184487] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15472.270170] alg: No test for adler32 (adler32-zlib) [15473.625954] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [15474.030847] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [15475.848259] Key type lgssc registered [15477.632658] Lustre: Echo OBD driver; http://www.lustre.org/ [15483.270193] LNet: Added LNI 192.168.201.106@tcp100 [8/256/0/180] [15484.398914] LNet: Added LNI 192.168.201.106@tcp101 [8/256/0/180] [15485.625029] LNet: Added LNI 192.168.201.106@tcp102 [8/256/0/180] [15486.818338] LNet: Added LNI 192.168.201.106@tcp103 [8/256/0/180] [15487.951210] LNet: Added LNI 192.168.201.106@tcp104 [8/256/0/180] [15489.194118] LNet: Added LNI 192.168.201.106@tcp105 [8/256/0/180] [15490.545934] LNet: Added LNI 192.168.201.106@tcp106 [8/256/0/180] [15491.461877] LNet: Added LNI 192.168.201.106@tcp107 [8/256/0/180] [15492.563811] LNet: Added LNI 192.168.201.106@tcp108 [8/256/0/180] [15493.514658] LNet: Added LNI 192.168.201.106@tcp109 [8/256/0/180] [15494.815539] LNet: Added LNI 192.168.201.106@tcp110 [8/256/0/180] [15496.067724] LNet: Added LNI 192.168.201.106@tcp111 [8/256/0/180] [15497.220554] LNet: Added LNI 192.168.201.106@tcp112 [8/256/0/180] [15498.418141] LNet: Added LNI 192.168.201.106@tcp113 [8/256/0/180] [15499.475122] LNet: Added LNI 192.168.201.106@tcp114 [8/256/0/180] [15500.655383] LNet: Added LNI 192.168.201.106@tcp115 [8/256/0/180] [15501.756024] LNet: Added LNI 192.168.201.106@tcp116 [8/256/0/180] [15502.759480] LNet: Added LNI 192.168.201.106@tcp117 [8/256/0/180] [15503.835304] LNet: Added LNI 192.168.201.106@tcp118 [8/256/0/180] [15505.050853] LNet: Added LNI 192.168.201.106@tcp119 [8/256/0/180] [15506.167386] LNet: Added LNI 192.168.201.106@tcp120 [8/256/0/180] [15507.299721] LNet: Added LNI 192.168.201.106@tcp121 [8/256/0/180] [15508.287945] LNet: Added LNI 192.168.201.106@tcp122 [8/256/0/180] [15509.399231] LNet: Added LNI 192.168.201.106@tcp123 [8/256/0/180] [15510.625441] LNet: Added LNI 192.168.201.106@tcp124 [8/256/0/180] [15511.686096] LNet: Added LNI 192.168.201.106@tcp125 [8/256/0/180] [15512.762013] LNet: Added LNI 192.168.201.106@tcp126 [8/256/0/180] [15513.768860] LNet: Added LNI 192.168.201.106@tcp127 [8/256/0/180] [15514.871829] LNet: Added LNI 192.168.201.106@tcp128 [8/256/0/180] [15516.009424] LNet: Added LNI 192.168.201.106@tcp129 [8/256/0/180] [15517.248945] LNet: Added LNI 192.168.201.106@tcp130 [8/256/0/180] [15518.317370] LNet: Added LNI 192.168.201.106@tcp131 [8/256/0/180] [15519.486870] LNet: Added LNI 192.168.201.106@tcp132 [8/256/0/180] [15520.587306] LNet: Added LNI 192.168.201.106@tcp133 [8/256/0/180] [15521.528354] LNet: Added LNI 192.168.201.106@tcp134 [8/256/0/180] [15522.550595] LNet: Added LNI 192.168.201.106@tcp135 [8/256/0/180] [15530.964229] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15531.003754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15532.172823] Lustre: 464340:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.106@tcp: can't add NID 192.168.201.106@tcp130: rc = -75 [15532.331170] LNetError: 461795:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15532.377428] LNetError: Protocol error connecting to 192.168.201.106@tcp120 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15532.406300] LNetError: Skipped 3 previous similar messages [15532.452024] LNet: 6 local NIs in recovery (showing 6): 192.168.201.106@tcp123, 192.168.201.106@tcp124, 192.168.201.106@tcp125, 192.168.201.106@tcp120, 192.168.201.106@tcp126, 192.168.201.106@tcp129 [15532.523348] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15532.605795] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15532.748851] Lustre: lustre-MDT0000: new disk, initializing [15532.833796] Lustre: 464397:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.106@tcp: can't add NID 192.168.201.106@tcp130: rc = -75 [15532.848197] Lustre: 464397:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15532.887993] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15532.908720] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15533.473790] LNetError: 461796:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15533.496699] LNetError: 461796:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 9 previous similar messages [15533.503401] LNetError: Protocol error connecting to 192.168.201.106@tcp111 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15533.517695] LNetError: Skipped 6 previous similar messages [15535.595804] LNetError: 464363:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15535.600918] LNetError: 464363:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 6 previous similar messages [15535.607748] LNetError: Protocol error connecting to 192.168.201.106@tcp123 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15535.613373] LNetError: Skipped 7 previous similar messages [15537.653675] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15539.686608] LNetError: 461797:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15539.706823] LNetError: 461797:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 4 previous similar messages [15539.721311] LNetError: Protocol error connecting to 192.168.201.106@tcp126 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15539.743624] LNetError: Skipped 4 previous similar messages [15547.875540] LNetError: 464368:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15547.895740] LNetError: 464368:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 1 previous similar message [15547.920508] LNetError: Protocol error connecting to 192.168.201.106@tcp126 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15550.399356] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15550.574925] Lustre: 464357:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15550.655720] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15550.674388] Lustre: Skipped 1 previous similar message [15550.803387] Lustre: lustre-MDT0001: new disk, initializing [15550.878190] Lustre: 465160:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.106@tcp: can't add NID 192.168.201.106@tcp130: rc = -75 [15550.897413] Lustre: 465160:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15551.113648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15551.189921] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15551.204043] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15557.729729] Lustre: 465385:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.106@tcp: can't add NID 192.168.201.106@tcp130: rc = -75 [15557.757539] Lustre: 465385:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15558.209429] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15563.239724] LNetError: 464374:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15563.269774] LNetError: Protocol error connecting to 192.168.201.106@tcp126 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15570.443685] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15576.997870] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15579.553393] LNetError: 464363:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15579.573596] LNetError: 464363:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 11 previous similar messages [15579.595034] LNetError: Protocol error connecting to 192.168.201.106@tcp127 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15579.623561] LNetError: Skipped 11 previous similar messages [15586.034147] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15586.310445] Lustre: lustre-OST0000: new disk, initializing [15586.316898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15586.324860] Lustre: 466335:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15586.433698] Lustre: 466340:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.106@tcp: can't add NID 192.168.201.106@tcp130: rc = -75 [15586.439338] Lustre: 466340:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [15586.507869] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15592.930787] LNet: 1 local NIs in recovery (showing 1): 192.168.201.106@tcp126 [15594.329132] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15596.102047] Lustre: 466760:0:(lustre_peer.c:124:class_add_uuid()) 192.168.201.106@tcp: can't add NID 192.168.201.106@tcp130: rc = -75 [15596.114360] Lustre: 466760:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [15596.239349] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15596.250377] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15596.461752] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15606.252883] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15615.400328] LNetError: 464372:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Bad magic(1) 0xacce7100 (0xeebc0ded expected) from 192.168.201.106 [15615.417321] LNetError: 464372:0:(socklnd_cb.c:1693:ksocknal_recv_hello()) Skipped 23 previous similar messages [15615.427225] LNetError: Protocol error connecting to 192.168.201.106@tcp129 at host 192.168.201.106:988: is it running a compatible version of Lustre? [15615.438433] LNetError: Skipped 23 previous similar messages [15616.993277] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [15617.001502] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15617.005669] LustreError: Skipped 1 previous similar message [15617.029332] Lustre: Skipped 1 previous similar message [15617.032762] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [15617.059425] Lustre: Skipped 1 previous similar message [15621.043694] Lustre: server umount lustre-OST0000 complete [15627.246606] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [15627.280250] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15627.289271] Lustre: Skipped 1 previous similar message [15632.354310] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [15632.357434] Lustre: Skipped 1 previous similar message [15632.873133] Lustre: server umount lustre-MDT0000 complete [15638.155443] LustreError: 464349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788985603 with bad export cookie 8891882289825920031 [15638.165724] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [15638.168864] LustreError: 464349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [15638.977455] Lustre: server umount lustre-MDT0001 complete [15653.137221] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [15656.431282] Key type lgssc unregistered [15656.944274] LNet: 468247:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15656.960785] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15656.966279] LNet: Removed LNI 192.168.201.106@tcp [15658.016177] LNet: Removed LNI 192.168.201.106@tcp100 [15658.018151] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15658.037803] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15658.054250] LNet: Removed LNI 192.168.201.106@tcp101 [15659.104395] LNet: Removed LNI 192.168.201.106@tcp102 [15659.122294] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15659.132472] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [15659.138533] LNet: Removed LNI 192.168.201.106@tcp103 [15660.192173] LNet: Removed LNI 192.168.201.106@tcp104 [15661.217394] LNet: Removed LNI 192.168.201.106@tcp105 [15661.223878] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15661.229695] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [15661.243102] LNet: Removed LNI 192.168.201.106@tcp106 [15662.304332] LNet: Removed LNI 192.168.201.106@tcp107 [15663.328330] LNet: Removed LNI 192.168.201.106@tcp108 [15664.352413] LNet: Removed LNI 192.168.201.106@tcp109 [15665.376306] LNet: Removed LNI 192.168.201.106@tcp110 [15665.416854] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15665.430924] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 4 previous similar messages [15665.453645] LNet: Removed LNI 192.168.201.106@tcp111 [15666.528205] LNet: Removed LNI 192.168.201.106@tcp112 [15667.552757] LNet: Removed LNI 192.168.201.106@tcp113 [15668.642068] LNet: Removed LNI 192.168.201.106@tcp114 [15669.667031] LNet: Removed LNI 192.168.201.106@tcp115 [15670.688261] LNet: Removed LNI 192.168.201.106@tcp116 [15671.712434] LNet: Removed LNI 192.168.201.106@tcp117 [15672.738411] LNet: Removed LNI 192.168.201.106@tcp118 [15673.761206] LNet: Removed LNI 192.168.201.106@tcp119 [15673.770867] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15673.785417] LNetError: 468247:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 8 previous similar messages [15673.793363] LNet: Removed LNI 192.168.201.106@tcp120 [15674.851669] LNet: Removed LNI 192.168.201.106@tcp121 [15675.872289] LNet: Removed LNI 192.168.201.106@tcp122 [15676.897089] LNet: Removed LNI 192.168.201.106@tcp123 [15677.984382] LNet: Removed LNI 192.168.201.106@tcp124 [15679.009573] LNet: Removed LNI 192.168.201.106@tcp125 [15679.015681] LNet: Removed LNI 192.168.201.106@tcp126 [15680.096298] LNet: Removed LNI 192.168.201.106@tcp127 [15681.136627] LNet: Removed LNI 192.168.201.106@tcp128 [15682.144473] LNet: Removed LNI 192.168.201.106@tcp129 [15682.150707] LNet: Removed LNI 192.168.201.106@tcp130 [15682.156786] LNet: Removed LNI 192.168.201.106@tcp131 [15682.164827] LNet: Removed LNI 192.168.201.106@tcp132 [15682.169697] LNet: Removed LNI 192.168.201.106@tcp133 [15682.174387] LNet: Removed LNI 192.168.201.106@tcp134 [15682.195404] LNet: Removed LNI 192.168.201.106@tcp135 [15683.627655] Key type .llcrypt unregistered [15683.638675] Key type ._llcrypt unregistered [15714.526763] Key type ._llcrypt registered [15714.529736] Key type .llcrypt registered [15714.657852] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [15731.194335] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [15732.726793] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15732.757779] alg: No test for adler32 (adler32-zlib) [15733.887784] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [15734.245586] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [15736.001373] Key type lgssc registered [15737.419958] Lustre: Echo OBD driver; http://www.lustre.org/ [15783.135519] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:29:06 (1788985746) [15807.147791] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [15810.844860] Key type lgssc unregistered [15811.092992] LNet: 472332:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [15811.099211] LNetError: 472332:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [15811.121521] LNet: Removed LNI 192.168.201.106@tcp [15812.216598] Key type .llcrypt unregistered [15812.218823] Key type ._llcrypt unregistered [15834.843485] Key type ._llcrypt registered [15834.844728] Key type .llcrypt registered [15835.001755] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [15836.236493] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [15836.276287] alg: No test for adler32 (adler32-zlib) [15838.088848] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [15838.596733] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [15840.481126] Key type lgssc registered [15842.610388] Lustre: Echo OBD driver; http://www.lustre.org/ [15854.002504] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [15854.042945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15855.373586] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [15855.432044] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [15855.503740] Lustre: lustre-MDT0000: new disk, initializing [15855.617685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [15855.639502] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [15861.137372] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15873.731848] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [15873.859471] Lustre: 473542:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [15873.979984] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [15873.988554] Lustre: Skipped 1 previous similar message [15874.095062] Lustre: lustre-MDT0001: new disk, initializing [15874.267701] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [15874.336574] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [15874.347196] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [15879.764162] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15890.838667] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [15898.502848] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [15907.994759] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [15908.473265] Lustre: lustre-OST0000: new disk, initializing [15908.487328] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [15908.496637] Lustre: 475512:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [15908.606183] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [15914.580152] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [15914.597955] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [15914.701907] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [15917.284200] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [15929.869161] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [15935.367801] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing lnet_if_list [15942.315271] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [15942.370610] LNet: Added LNI 192.168.201.106@tcp10 [8/256/0/180] [15950.570926] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [15950.617747] LNet: Added LNI 192.168.201.106@tcp11 [8/256/0/180] [15955.318712] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [15955.383698] LNet: Added LNI 192.168.201.106@tcp12 [8/256/0/180] [15961.900344] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [15961.964742] LNet: Added LNI 192.168.201.106@tcp13 [8/256/0/180] [15967.576458] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [15967.622416] LNet: Added LNI 192.168.201.106@tcp14 [8/256/0/180] [15973.340460] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [15973.394036] LNet: Added LNI 192.168.201.106@tcp15 [8/256/0/180] [15979.587925] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [15979.634121] LNet: Added LNI 192.168.201.106@tcp16 [8/256/0/180] [15986.166355] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [15986.188303] LNet: Added LNI 192.168.201.106@tcp17 [8/256/0/180] [15991.681650] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [15991.723213] LNet: Added LNI 192.168.201.106@tcp18 [8/256/0/180] [15997.588635] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [15997.624858] LNet: Added LNI 192.168.201.106@tcp19 [8/256/0/180] [16003.347827] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [16003.393986] LNet: Added LNI 192.168.201.106@tcp20 [8/256/0/180] [16009.645522] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [16009.667856] LNet: Added LNI 192.168.201.106@tcp21 [8/256/0/180] [16015.488894] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [16015.524739] LNet: Added LNI 192.168.201.106@tcp22 [8/256/0/180] [16022.363575] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [16022.405424] LNet: Added LNI 192.168.201.106@tcp23 [8/256/0/180] [16029.374743] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [16029.399798] LNet: Added LNI 192.168.201.106@tcp24 [8/256/0/180] [16036.098836] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [16036.134174] LNet: Added LNI 192.168.201.106@tcp25 [8/256/0/180] [16043.580387] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [16043.631708] LNet: Added LNI 192.168.201.106@tcp26 [8/256/0/180] [16051.661976] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [16051.681209] LNet: Added LNI 192.168.201.106@tcp27 [8/256/0/180] [16059.559425] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [16059.610422] LNet: Added LNI 192.168.201.106@tcp28 [8/256/0/180] [16067.197834] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [16067.227318] LNet: Added LNI 192.168.201.106@tcp29 [8/256/0/180] [16074.582412] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [16074.623629] LNet: Added LNI 192.168.201.106@tcp30 [8/256/0/180] [16082.448596] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [16082.495975] LNet: Added LNI 192.168.201.106@tcp31 [8/256/0/180] [16090.035891] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [16090.081246] LNet: Added LNI 192.168.201.106@tcp32 [8/256/0/180] [16096.874663] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [16096.899751] LNet: Added LNI 192.168.201.106@tcp33 [8/256/0/180] [16104.401460] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [16104.430336] LNet: Added LNI 192.168.201.106@tcp34 [8/256/0/180] [16112.155473] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [16112.186512] LNet: Added LNI 192.168.201.106@tcp35 [8/256/0/180] [16120.706573] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [16120.743419] LNet: Added LNI 192.168.201.106@tcp36 [8/256/0/180] [16129.505739] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [16129.579724] LNet: Added LNI 192.168.201.106@tcp37 [8/256/0/180] [16139.680601] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [16139.748050] LNet: Added LNI 192.168.201.106@tcp38 [8/256/0/180] [16149.740893] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [16149.761821] LNet: Added LNI 192.168.201.106@tcp39 [8/256/0/180] [16157.815536] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [16157.867169] LNet: Added LNI 192.168.201.106@tcp40 [8/256/0/180] [16165.794929] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [16165.817332] LNet: Added LNI 192.168.201.106@tcp41 [8/256/0/180] [16173.311677] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [16173.348124] LNet: Added LNI 192.168.201.106@tcp42 [8/256/0/180] [16182.835288] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [16182.859041] LNet: Added LNI 192.168.201.106@tcp43 [8/256/0/180] [16193.661694] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [16193.713629] LNet: Added LNI 192.168.201.106@tcp44 [8/256/0/180] [16204.838296] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [16204.861733] LNet: Added LNI 192.168.201.106@tcp45 [8/256/0/180] [16213.032296] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [16213.081142] LNet: Added LNI 192.168.201.106@tcp46 [8/256/0/180] [16222.137588] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [16222.205154] LNet: Added LNI 192.168.201.106@tcp47 [8/256/0/180] [16231.161207] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [16231.201509] LNet: Added LNI 192.168.201.106@tcp48 [8/256/0/180] [16240.271331] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [16240.294328] LNet: Added LNI 192.168.201.106@tcp49 [8/256/0/180] [16249.190601] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [16249.237624] LNet: Added LNI 192.168.201.106@tcp50 [8/256/0/180] [16273.382219] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16273.387338] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16273.409021] Lustre: Skipped 1 previous similar message [16273.421902] Lustre: Skipped 2 previous similar messages [16276.838928] Lustre: server umount lustre-MDT0000 complete [16281.503613] LustreError: 473535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788986247 with bad export cookie 7995353002375193529 [16281.513354] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16281.519595] LustreError: 473535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [16281.959762] Lustre: server umount lustre-MDT0001 complete [16299.937133] Lustre: 472813:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788986249/real 1788986249] req@00000000cd37404c x1875887578605568/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788986265 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16299.956455] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16299.969502] Lustre: Skipped 1 previous similar message [16303.074060] Lustre: 472816:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788986252/real 1788986252] req@000000008103b876 x1875887578605696/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788986268 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16304.273810] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [16305.121088] Lustre: 472816:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788986254/real 1788986254] req@0000000055dcaace x1875887578605824/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788986270 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16308.128116] Lustre: 472813:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788986257/real 1788986257] req@000000007540d4cc x1875887578606080/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788986273 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16313.312821] Lustre: 472814:0:(client.c:2504:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788986262/real 1788986262] req@0000000091ddee23 x1875887578606208/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788986278 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [16314.819185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16326.625435] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0x6ef53331c86d2e40 [16326.639922] Lustre: MGC192.168.201.106@tcp: Connection restored to 0@lo (at 0@lo) [16327.062907] LustreError: 475506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16327.077896] LustreError: 475506:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 4 previous similar messages [16327.155157] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16329.121091] LustreError: 475507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16330.148442] LustreError: 475506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16330.156477] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16333.384391] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [16335.339553] LustreError: 475507:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16340.451851] LustreError: 475506:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16343.803793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16344.263757] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16345.342858] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [16349.977638] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [16360.116859] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16368.399597] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16390.627181] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [16390.648133] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16390.677406] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16396.250949] Lustre: server umount lustre-OST0000 complete [16400.878771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16400.890674] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16400.910278] Lustre: Skipped 1 previous similar message [16400.922563] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16400.930753] Lustre: Skipped 2 previous similar messages [16406.555645] Lustre: server umount lustre-MDT0000 complete [16407.011316] LustreError: 499771:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16407.023281] LustreError: 499771:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16410.381062] LustreError: 473535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788986376 with bad export cookie 7995353002375196224 [16410.391595] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16410.392522] LustreError: 473535:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16410.742355] Lustre: server umount lustre-MDT0001 complete [16424.310715] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [16427.321930] Key type lgssc unregistered [16427.790477] LNet: 502569:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16427.797174] LNetError: 502569:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16427.802907] LNet: Removed LNI 192.168.201.106@tcp [16427.809884] LNet: Removed LNI 192.168.201.106@tcp10 [16427.812769] LNet: Removed LNI 192.168.201.106@tcp11 [16427.816401] LNet: Removed LNI 192.168.201.106@tcp12 [16427.819284] LNet: Removed LNI 192.168.201.106@tcp13 [16427.822293] LNet: Removed LNI 192.168.201.106@tcp14 [16427.831754] LNet: Removed LNI 192.168.201.106@tcp15 [16427.835332] LNet: Removed LNI 192.168.201.106@tcp16 [16427.838705] LNet: Removed LNI 192.168.201.106@tcp17 [16427.841896] LNet: Removed LNI 192.168.201.106@tcp18 [16427.846572] LNet: Removed LNI 192.168.201.106@tcp19 [16427.851373] LNet: Removed LNI 192.168.201.106@tcp20 [16427.857510] LNet: Removed LNI 192.168.201.106@tcp21 [16427.861749] LNet: Removed LNI 192.168.201.106@tcp22 [16427.863517] LNet: Removed LNI 192.168.201.106@tcp23 [16427.870396] LNet: Removed LNI 192.168.201.106@tcp24 [16427.872548] LNet: Removed LNI 192.168.201.106@tcp25 [16427.878480] LNet: Removed LNI 192.168.201.106@tcp26 [16427.891547] LNet: Removed LNI 192.168.201.106@tcp27 [16427.896733] LNet: Removed LNI 192.168.201.106@tcp28 [16427.900839] LNet: Removed LNI 192.168.201.106@tcp29 [16427.907123] LNet: Removed LNI 192.168.201.106@tcp30 [16427.909179] LNet: Removed LNI 192.168.201.106@tcp31 [16427.912689] LNet: Removed LNI 192.168.201.106@tcp32 [16427.920609] LNet: Removed LNI 192.168.201.106@tcp33 [16427.923857] LNet: Removed LNI 192.168.201.106@tcp34 [16427.930794] LNet: Removed LNI 192.168.201.106@tcp35 [16427.939144] LNet: Removed LNI 192.168.201.106@tcp36 [16427.947949] LNet: Removed LNI 192.168.201.106@tcp37 [16427.954522] LNet: Removed LNI 192.168.201.106@tcp38 [16427.959382] LNet: Removed LNI 192.168.201.106@tcp39 [16427.961684] LNet: Removed LNI 192.168.201.106@tcp40 [16427.974866] LNet: Removed LNI 192.168.201.106@tcp41 [16427.977894] LNet: Removed LNI 192.168.201.106@tcp42 [16427.984860] LNet: Removed LNI 192.168.201.106@tcp43 [16427.989511] LNet: Removed LNI 192.168.201.106@tcp44 [16427.991457] LNet: Removed LNI 192.168.201.106@tcp45 [16428.005961] LNet: Removed LNI 192.168.201.106@tcp46 [16428.011277] LNet: Removed LNI 192.168.201.106@tcp47 [16428.015123] LNet: Removed LNI 192.168.201.106@tcp48 [16428.020181] LNet: Removed LNI 192.168.201.106@tcp49 [16428.029540] LNet: Removed LNI 192.168.201.106@tcp50 [16428.951497] Key type .llcrypt unregistered [16428.957328] Key type ._llcrypt unregistered [16451.971084] Key type ._llcrypt registered [16451.972449] Key type .llcrypt registered [16452.175635] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [16468.400608] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [16469.581808] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16469.630264] alg: No test for adler32 (adler32-zlib) [16470.745400] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [16470.989774] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [16472.632233] Key type lgssc registered [16473.792761] Lustre: Echo OBD driver; http://www.lustre.org/ [16517.798838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16517.830609] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16519.311209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [16519.355817] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [16519.467983] Lustre: lustre-MDT0000: new disk, initializing [16519.565628] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16519.611479] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16525.223839] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [16537.155510] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16537.346658] Lustre: 505766:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [16537.401755] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [16537.406159] Lustre: Skipped 1 previous similar message [16537.498441] Lustre: lustre-MDT0001: new disk, initializing [16537.616310] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16537.647696] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [16537.667840] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [16543.078590] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [16553.332413] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16560.036550] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16568.670716] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16568.910915] Lustre: *** cfs_fail_loc=90f, val=0*** [16568.961286] Lustre: lustre-OST0000: new disk, initializing [16568.964343] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [16568.968744] Lustre: 507786:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16569.054282] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16576.532898] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [16578.598813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [16578.613761] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [16578.765848] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [16588.117630] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16599.526209] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16599.526782] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16599.550746] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16599.560876] Lustre: Skipped 1 previous similar message [16603.634758] Lustre: server umount lustre-OST0000 complete [16609.252169] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [16609.262142] LustreError: Skipped 1 previous similar message [16609.269729] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16609.294967] Lustre: Skipped 1 previous similar message [16609.306483] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16614.907827] Lustre: server umount lustre-MDT0000 complete [16619.791610] LustreError: 505759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788986585 with bad export cookie 975931349690011452 [16619.803912] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16619.808901] LustreError: 505759:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [16620.293909] Lustre: server umount lustre-MDT0001 complete [16634.138833] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [16637.220413] Key type lgssc unregistered [16637.557661] LNet: 509689:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16637.570150] LNetError: 509689:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16637.594294] LNet: Removed LNI 192.168.201.106@tcp [16638.744418] Key type .llcrypt unregistered [16638.749046] Key type ._llcrypt unregistered [16662.337655] Key type ._llcrypt registered [16662.342587] Key type .llcrypt registered [16662.483816] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [16663.900592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16663.909066] alg: No test for adler32 (adler32-zlib) [16664.899626] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [16665.141970] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [16666.833788] Key type lgssc registered [16667.886335] Lustre: Echo OBD driver; http://www.lustre.org/ [16676.864154] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [16676.876830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16678.265558] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16683.227950] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [16687.591821] LustreError: 510959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16687.611738] LustreError: 510959:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 2 previous similar messages [16692.708972] LustreError: 510958:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16693.503446] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16693.846539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [16699.515319] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [16708.550507] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [16716.089627] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [16725.125417] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16725.524673] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16734.269636] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [16743.805176] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [16761.884223] loop0: detected capacity change from 0 to 204800000 [16762.112426] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16762.128656] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16762.147878] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16768.743755] loop0: detected capacity change from 0 to 204800000 [16768.925876] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16768.934897] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16768.939992] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [16775.057374] loop0: detected capacity change from 0 to 204800000 [16782.393232] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [16782.626445] Lustre: fs15246-OST0000: new disk, initializing [16782.633443] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [16782.640972] Lustre: 514576:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [16782.734900] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [16789.547362] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [16799.977942] loop1: detected capacity change from 0 to 204800000 [16807.836604] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [16808.150058] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [16808.267514] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [16808.469238] Lustre: fs15246-MDT0000: new disk, initializing [16808.814066] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [16808.888610] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [16809.095177] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [16810.145655] Lustre: 515697:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@0000000043801f16 x1875888444591488/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788986791 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [16815.484335] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [16819.314711] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [16819.329579] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [16819.384953] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [16841.190690] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16841.220544] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16846.311845] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [16847.070145] Lustre: server umount fs15246-MDT0000 complete [16851.430879] LustreError: 510953:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16856.600162] LustreError: 510954:0:(ldlm_lib.c:1199:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16858.757323] Lustre: server umount fs15246-OST0000 complete [16876.513822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [16876.517235] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16876.523775] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16882.156395] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16882.165822] Lustre: Skipped 3 previous similar messages [16887.274719] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [16887.290228] Lustre: Skipped 1 previous similar message [16890.338843] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16890.523974] Lustre: server umount lustre-OST0000 complete [16897.533257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [16897.561158] Lustre: Skipped 1 previous similar message [16897.576407] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [16897.594603] Lustre: Skipped 1 previous similar message [16910.305669] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [16910.559042] Lustre: server umount lustre-MDT0000 complete [16912.876154] LustreError: 510959:0:(ldlm_lib.c:1199:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [16912.903223] LustreError: 510959:0:(ldlm_lib.c:1199:target_handle_connect()) Skipped 1 previous similar message [16915.221299] LustreError: 510938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788986880 with bad export cookie 15929174246329871919 [16915.245340] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [16915.261527] LustreError: 510938:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [16915.652724] Lustre: server umount lustre-MDT0001 complete [16929.660782] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [16933.465379] Key type lgssc unregistered [16933.822337] LNet: 518797:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [16933.844937] LNetError: 518797:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [16933.885654] LNet: Removed LNI 192.168.201.106@tcp [16935.207396] Key type .llcrypt unregistered [16935.210393] Key type ._llcrypt unregistered [16961.605560] Key type ._llcrypt registered [16961.608947] Key type .llcrypt registered [16961.793491] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [16963.217522] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [16963.498428] alg: No test for adler32 (adler32-zlib) [16964.641990] Lustre: Lustre: Build Version: 2.17.58_42_gf11d80b [16964.895239] LNet: Added LNI 192.168.201.106@tcp [8/256/0/180] [16966.600651] Key type lgssc registered [16968.374567] Lustre: Echo OBD driver; http://www.lustre.org/ [17021.176412] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [17038.400481] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [17089.203523] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [17089.264108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17090.728587] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [17090.800342] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [17090.899394] Lustre: lustre-MDT0000: new disk, initializing [17090.993388] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [17091.020430] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [17096.837638] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [17112.660566] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [17112.860727] Lustre: 523292:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [17113.005103] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [17113.014882] Lustre: Skipped 1 previous similar message [17113.173991] Lustre: lustre-MDT0001: new disk, initializing [17113.508627] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [17113.582654] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [17113.604437] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [17120.068685] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [17135.961585] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [17146.140412] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [17157.983685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [17158.342927] Lustre: lustre-OST0000: new disk, initializing [17158.347974] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [17158.367759] Lustre: 525264:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [17158.502954] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [17164.957757] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [17164.973194] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [17165.182361] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [17168.213499] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_default_debug -1 all [17180.468588] Lustre: DEBUG MARKER: oleg106-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [17187.737982] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [17188.090241] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [17193.997635] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [17194.327333] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [17199.029899] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17199.057031] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17204.195890] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17204.210610] Lustre: Skipped 2 previous similar messages [17209.313958] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [17209.331983] Lustre: Skipped 1 previous similar message [17210.849311] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17211.297150] Lustre: server umount lustre-OST0000 complete [17219.559417] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [17219.583383] Lustre: Skipped 1 previous similar message [17219.598149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17224.674166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [17224.676909] Lustre: Skipped 1 previous similar message [17231.328179] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [17231.666102] Lustre: server umount lustre-MDT0000 complete [17237.067762] LustreError: 523282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788987202 with bad export cookie 8615950894440445351 [17237.077357] LustreError: MGC192.168.201.106@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [17237.081674] LustreError: 523282:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [17237.681321] Lustre: server umount lustre-MDT0001 complete [17277.077505] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing set_hostid [17294.970315] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing load_modules_local [17338.095870] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 16990 sec ========= 16:55:00 (1788987300) [17341.473622] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 16:55:03 (1788987303) === [17348.000433] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 16:55:10 (1788987310) === [17408.999494] Lustre: DEBUG MARKER: oleg106-server.virtnet: executing unload_modules_local [17413.238656] Key type lgssc unregistered [17413.781162] LNet: 531935:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [17413.790700] LNetError: 531935:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [17413.815905] LNet: Removed LNI 192.168.201.106@tcp [17414.664942] Key type .llcrypt unregistered [17414.674137] Key type ._llcrypt unregistered