[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 450970719 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003220] x2apic enabled [ 0.004011] Switched APIC routing to physical x2apic. [ 0.005016] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009012] pid_max: default: 32768 minimum: 301 [ 0.011087] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013037] SELinux: Initializing. [ 0.014086] *** VALIDATE selinux *** [ 0.022504] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026781] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.028049] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029116] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.031060] *** VALIDATE tmpfs *** [ 0.032474] *** VALIDATE proc *** [ 0.033178] *** VALIDATE cgroup *** [ 0.034010] *** VALIDATE cgroup2 *** [ 0.036261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.038097] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.039006] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.040030] Spectre V2 : User space: Vulnerable [ 0.041005] Speculative Store Bypass: Vulnerable [ 0.043799] debug: unmapping init [mem 0xffffffff94a59000-0xffffffff94a60fff] [ 0.046000] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.046754] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.047039] ... version: 2 [ 0.048013] ... bit width: 48 [ 0.049012] ... generic registers: 4 [ 0.050014] ... value mask: 0000ffffffffffff [ 0.051017] ... max period: 00007fffffffffff [ 0.052017] ... fixed-purpose events: 3 [ 0.053020] ... event mask: 000000070000000f [ 0.054333] rcu: Hierarchical SRCU implementation. [ 0.056550] smp: Bringing up secondary CPUs ... [ 0.057596] x86: Booting SMP configuration: [ 0.058026] .... node #0, CPUs: #1 #2 #3 [ 0.061069] smp: Brought up 1 node, 4 CPUs [ 0.063016] smpboot: Max logical packages: 1 [ 0.064013] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.103632] node 0 deferred pages initialised in 36ms [ 0.106164] devtmpfs: initialized [ 0.107152] x86/mm: Memory block size: 128MB [ 0.111262] gcov: version magic: 0x41383552 [ 0.114671] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.117062] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.118336] pinctrl core: initialized pinctrl subsystem [ 0.120118] [ 0.120398] ************************************************************* [ 0.121017] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.123009] ** ** [ 0.124022] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.125010] ** ** [ 0.127008] ** This means that this kernel is built to expose internal ** [ 0.128006] ** IOMMU data structures, which may compromise security on ** [ 0.129007] ** your system. ** [ 0.131008] ** ** [ 0.132007] ** If you see this message and you are not debugging the ** [ 0.133007] ** kernel, report this immediately to your vendor! ** [ 0.134008] ** ** [ 0.136007] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.137007] ************************************************************* [ 0.138553] NET: Registered protocol family 16 [ 0.140314] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.141049] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.143041] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.145372] cpuidle: using governor menu [ 0.148224] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.151457] PCI: Using configuration type 1 for base access [ 0.154126] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.164064] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.165025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.169070] cryptd: max_cpu_qlen set to 1000 [ 0.171214] ACPI: Added _OSI(Module Device) [ 0.172011] ACPI: Added _OSI(Processor Device) [ 0.172974] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.174009] ACPI: Added _OSI(Processor Aggregator Device) [ 0.177123] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.182083] ACPI: Interpreter enabled [ 0.182956] ACPI: PM: (supports S0 S3 S4 S5) [ 0.184010] ACPI: Using IOAPIC for interrupt routing [ 0.185081] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.187335] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.194421] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.196039] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.198016] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.200145] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.207209] acpiphp: Slot [2] registered [ 0.209349] acpiphp: Slot [5] registered [ 0.211307] acpiphp: Slot [6] registered [ 0.213123] acpiphp: Slot [7] registered [ 0.215128] acpiphp: Slot [8] registered [ 0.217135] acpiphp: Slot [9] registered [ 0.219145] acpiphp: Slot [10] registered [ 0.221129] acpiphp: Slot [3] registered [ 0.223137] acpiphp: Slot [4] registered [ 0.226183] acpiphp: Slot [11] registered [ 0.227143] acpiphp: Slot [12] registered [ 0.229245] acpiphp: Slot [13] registered [ 0.230250] acpiphp: Slot [14] registered [ 0.232119] acpiphp: Slot [15] registered [ 0.234092] acpiphp: Slot [16] registered [ 0.235114] acpiphp: Slot [17] registered [ 0.236129] acpiphp: Slot [18] registered [ 0.238104] acpiphp: Slot [19] registered [ 0.239123] acpiphp: Slot [20] registered [ 0.241111] acpiphp: Slot [21] registered [ 0.242116] acpiphp: Slot [22] registered [ 0.244108] acpiphp: Slot [23] registered [ 0.245083] acpiphp: Slot [24] registered [ 0.247099] acpiphp: Slot [25] registered [ 0.248109] acpiphp: Slot [26] registered [ 0.249048] acpiphp: Slot [27] registered [ 0.250044] acpiphp: Slot [28] registered [ 0.250974] acpiphp: Slot [29] registered [ 0.252080] acpiphp: Slot [30] registered [ 0.253025] acpiphp: Slot [31] registered [ 0.253953] PCI host bridge to bus 0000:00 [ 0.255012] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.256014] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.258014] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.259015] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.260018] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.263021] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.264272] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.266702] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.269988] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.277517] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.282444] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.285017] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.286000] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.288019] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.291685] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.295065] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.298046] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.300709] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.306013] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.319660] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.324019] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.332694] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.340018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.348021] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.373019] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.383268] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.393017] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.401017] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.425030] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.439537] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.445014] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.451022] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.474031] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.494412] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.503032] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.510025] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.531027] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.546519] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.560019] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.571019] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.587018] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.599056] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.607025] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.621026] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.648031] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.663035] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.666286] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.669263] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.671322] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.674222] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.680034] iommu: Default domain type: Passthrough [ 0.681332] SCSI subsystem initialized [ 0.682167] ACPI: bus type USB registered [ 0.684119] usbcore: registered new interface driver usbfs [ 0.686113] usbcore: registered new interface driver hub [ 0.688123] usbcore: registered new device driver usb [ 0.690271] pps_core: LinuxPPS API ver. 1 registered [ 0.692023] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.697092] PTP clock support registered [ 0.699118] EDAC MC: Ver: 3.0.0 [ 0.701137] PCI: Using ACPI for IRQ routing [ 0.703869] NetLabel: Initializing [ 0.705014] NetLabel: domain hash size = 128 [ 0.706012] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.708098] NetLabel: unlabeled traffic allowed by default [ 0.711157] vgaarb: loaded [ 0.712246] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.714015] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.721190] clocksource: Switched to clocksource kvm-clock [ 0.839227] VFS: Disk quotas dquot_6.6.0 [ 0.840693] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.843323] *** VALIDATE ramfs *** [ 0.844454] *** VALIDATE hugetlbfs *** [ 0.845918] pnp: PnP ACPI init [ 0.848417] pnp: PnP ACPI: found 6 devices [ 0.864978] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.868484] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.870885] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.873176] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.875731] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.878587] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.881471] NET: Registered protocol family 2 [ 0.884425] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.889265] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.892832] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.898277] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.901768] TCP: Hash tables configured (established 65536 bind 65536) [ 0.904368] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.907424] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.909614] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.913181] NET: Registered protocol family 1 [ 0.915737] RPC: Registered named UNIX socket transport module. [ 0.918345] RPC: Registered udp transport module. [ 0.920097] RPC: Registered tcp transport module. [ 0.921897] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.924342] NET: Registered protocol family 44 [ 0.926061] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.928451] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.930725] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.933129] PCI: CLS 0 bytes, default 64 [ 0.934900] Unpacking initramfs... [ 2.353725] debug: unmapping init [mem 0xffff92b9bcc54000-0xffff92b9bffbffff] [ 2.357969] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.361267] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.363864] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.895853] Initialise system trusted keyrings [ 2.897817] Key type blacklist registered [ 2.900093] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.909734] zbud: loaded [ 2.912885] *** VALIDATE nfs *** [ 2.914274] *** VALIDATE nfs4 *** [ 2.915917] pstore: using deflate compression [ 2.919880] Platform Keyring initialized [ 3.021720] NET: Registered protocol family 38 [ 3.023288] Key type asymmetric registered [ 3.024583] Asymmetric key parser 'x509' registered [ 3.026106] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 3.028634] io scheduler mq-deadline registered [ 3.030772] io scheduler kyber registered [ 3.031940] io scheduler bfq registered [ 3.035142] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 3.037364] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 3.040388] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 3.043304] ACPI: Power Button [PWRF] [ 3.049299] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 3.056433] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 3.070686] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.078619] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.095101] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.123338] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.152717] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.159219] Non-volatile memory driver v1.3 [ 3.161175] Linux agpgart interface v0.103 [ 3.194272] virtio_blk virtio1: [vda] 134864 512-byte logical blocks (69.1 MB/65.9 MiB) [ 3.197462] vda: detected capacity change from 0 to 69050368 [ 3.212186] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.215292] vdb: detected capacity change from 0 to 1073741824 [ 3.229634] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.232559] vdc: detected capacity change from 0 to 2621440000 [ 3.245388] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.248461] vdd: detected capacity change from 0 to 2621440000 [ 3.264511] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.268312] vde: detected capacity change from 0 to 4294967296 [ 3.291689] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.294518] vdf: detected capacity change from 0 to 4294967296 [ 3.305833] libphy: Fixed MDIO Bus: probed [ 3.311973] usbcore: registered new interface driver usbserial_generic [ 3.314245] usbserial: USB Serial support registered for generic [ 3.316443] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.320591] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.322330] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.324280] mousedev: PS/2 mouse device common for all mice [ 3.327557] rtc_cmos 00:05: RTC can wake from S4 [ 3.335043] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.335365] rtc_cmos 00:05: registered as rtc0 [ 3.340051] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.341204] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.342864] intel_pstate: CPU model not supported [ 3.347246] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.352935] hid: raw HID events driver (C) Jiri Kosina [ 3.354705] usbcore: registered new interface driver usbhid [ 3.356293] usbhid: USB HID core driver [ 3.357541] drop_monitor: Initializing network drop monitor service [ 3.359524] Initializing XFRM netlink socket [ 3.361190] NET: Registered protocol family 10 [ 3.364097] Segment Routing with IPv6 [ 3.365723] NET: Registered protocol family 17 [ 3.367322] mpls_gso: MPLS GSO support [ 3.373444] RAS: Correctable Errors collector initialized. [ 3.375666] AVX version of gcm_enc/dec engaged. [ 3.377423] AES CTR mode by8 optimization enabled [ 3.460605] sched_clock: Marking stable (3460579080, 0)->(4301107963, -840528883) [ 3.464724] registered taskstats version 1 [ 3.466736] Loading compiled-in X.509 certificates [ 3.468784] zswap: loaded using pool lzo/zbud [ 3.496533] Key type big_key registered [ 3.508818] Key type encrypted registered [ 3.510627] ima: No TPM chip found, activating TPM-bypass! [ 3.513062] ima: Allocated hash algorithm: sha1 [ 3.515151] ima: No architecture policies found [ 3.517264] evm: Initialising EVM extended attributes: [ 3.519408] evm: security.selinux [ 3.520779] evm: security.ima [ 3.521915] evm: security.capability [ 3.523072] evm: HMAC attrs: 0x1 [ 3.525223] rtc_cmos 00:05: setting system clock to 2026-06-01 17:35:32 UTC (1780335332) [ 3.530703] debug: unmapping init [mem 0xffffffff95a03000-0xffffffff95bfffff] [ 3.533285] debug: unmapping init [mem 0xffffffff94782000-0xffffffff94a58fff] [ 3.542152] Write protecting the kernel read-only data: 28672k [ 3.546134] debug: unmapping init [mem 0xffffffff92e03000-0xffffffff92ffffff] [ 3.548579] debug: unmapping init [mem 0xffffffff93714000-0xffffffff937fffff] [ 3.584000] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.593093] systemd[1]: Detected virtualization kvm. [ 3.595120] systemd[1]: Detected architecture x86-64. [ 3.596799] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.625139] systemd[1]: No hostname configured. [ 3.627069] systemd[1]: Set hostname to . [ 3.629296] random: systemd: uninitialized urandom read (16 bytes read) [ 3.631851] systemd[1]: Initializing machine ID from random generator. [ 3.687074] random: ln: uninitialized urandom read (6 bytes read) [ 3.778169] random: systemd: uninitialized urandom read (16 bytes read) [ 3.780626] systemd[1]: Reached target Swap. [ OK ] Reached target Swap. [ 3.784712] systemd[1]: Started Dispatch Password Requests to Console Directory Watch. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ 3.790949] systemd[1]: Reached target Local Encrypted Volumes. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Paths. [ OK ] Listening on Journal Socket. Starting Setup Virtual Console... [ OK ] Started Memstrack Anylazing Service. Starting Create list of required st…ce nodes for the current kernel... Starting Apply Kernel Variables... [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Slices. [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Listening on Journal Socket (/dev/log). [ OK ] Reached target Sockets. Starting Journal Service... [ OK ] Started Setup Virtual Console. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting Create Static Device Nodes in /dev... Starting dracut cmdline hook... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.417107] device-mapper: uevent: version 1.0.3 [ 4.419500] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 5.069324] virtio_net virtio0 ens2: renamed from eth0 [ 5.076079] random: fast init done [ 5.112245] scsi host0: ata_piix [ 5.145880] scsi host1: ata_piix [ 5.147430] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.149901] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.819472] dracut-initqueue[584]: RTNETLINK answers: File exists [ 10.027746] random: crng init done [ 10.029992] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.383767] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped target Timers. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Sockets. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Swap. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped target Local File Systems. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped Apply Kernel Variables. Stopping udev Kernel Device Manager... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Kernel Socket. [ OK ] Closed udev Control Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.589487] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.817717] SELinux: Disabled at runtime. [ 11.882757] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.891177] systemd[1]: Detected virtualization kvm. [ 11.892718] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.318788] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.323737] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.331873] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.335156] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.337325] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.342704] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.348521] systemd[1]: Starting Remount Root and Kernel File Systems... Starting Remount Root and Kernel File Systems... [ OK ] Reached target rpc_pipefs.target. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Stopped target Switch Root. [ OK ] Listening on Process Core Dump Socket. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Created slice system-getty.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Apply Kernel Variables... [ OK ] Listening on udev Control Socket. Starting udev Coldplug all Devices... [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. Activating swap /dev/disk/by-label/SWAP... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on initctl Compatibility Named Pipe. Mounting POSIX Message Queue File System... [ 12.473569] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS Mounting Huge Pages File System... Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd Root File System. [ OK ] Stopped target Initrd File Systems. [ OK ] Created slice User and Session Slice. [ OK ] Reached target Slices. [ OK ] Started Journal Service. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Kernel Debug File System. [ OK ] Started Apply Kernel Variables. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started udev Coldplug all Devices. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.778304] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 13.095168] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.159921] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.276938] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.286705] EDAC sbridge: Ver: 1.1.2 [ 14.641143] Key type dns_resolver registered [ 14.963283] NFS: Registering the id_resolver key type [ 14.965299] Key type id_resolver registered [ 14.966842] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting RPC Bind... Starting Update UTMP about System Boot/Shutdown... [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started RPC Bind. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started D-Bus System Message Bus. Starting Login Service... [ OK ] Reached target sshd-keygen.target. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. Starting Network Manager Wait Online... [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... Starting Hostname Service... [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Started Command Scheduler. [ OK ] Started Getty on tty1. [ OK ] Reached target Login Prompts. [ OK ] Started OpenSSH server daemon. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg207-server login: [ 42.863746] libcfs: loading out-of-tree module taints kernel. [ 42.904273] Key type ._llcrypt registered [ 42.907547] Key type .llcrypt registered [ 43.014838] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 65.427532] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 67.219489] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 67.242603] alg: No test for adler32 (adler32-zlib) [ 68.731624] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 69.581411] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 71.464298] Key type lgssc registered [ 73.430142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 94.801802] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 141.287314] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 154.963258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 155.019839] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 156.231209] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 156.261117] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 156.341173] Lustre: lustre-MDT0000: new disk, initializing [ 156.413670] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 156.431506] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 161.179140] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 175.331972] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 175.539284] Lustre: 6518:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 175.626204] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 175.636965] Lustre: Skipped 1 previous similar message [ 175.861347] Lustre: lustre-MDT0001: new disk, initializing [ 175.997352] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 176.056173] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 176.068180] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 181.466224] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 187.336759] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 199.130482] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 199.442921] Lustre: lustre-OST0000: new disk, initializing [ 199.446739] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 199.465454] Lustre: 8425:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 199.550171] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 200.715726] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 200.725834] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 200.844383] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 204.709332] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 219.929566] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 220.042702] Lustre: lustre-OST0001: new disk, initializing [ 220.047681] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 220.056380] Lustre: 9479:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 220.115365] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 226.810940] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 228.390236] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 228.401299] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 228.520723] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 239.241879] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 247.611534] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 252.904307] hrtimer: interrupt took 4011081 ns [ 254.650671] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing check_logdir /tmp/testlogs/ [ 259.749461] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing yml_node [ 264.665518] Lustre: DEBUG MARKER: Client: 2.17.53.34 [ 267.769854] Lustre: DEBUG MARKER: MDS: 2.17.53.34 [ 270.899575] Lustre: DEBUG MARKER: OSS: 2.17.53.34 [ 272.698418] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Jun 1 13:40:00 EDT 2026 [ 291.238632] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 292.730484] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 309.218482] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 309.233400] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 309.249313] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.246865] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 310.250147] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 310.269123] Lustre: Skipped 3 previous similar messages [ 313.686155] Lustre: server umount lustre-MDT0000 complete [ 317.450151] LustreError: 8424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335646 with bad export cookie 13467849139632486355 [ 317.454652] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 317.464948] LustreError: 8424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 320.485514] LustreError: 7667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 320.490034] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 320.500549] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 320.513033] LustreError: 7667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 320.553281] Lustre: Skipped 3 previous similar messages [ 322.529742] LustreError: 6523:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 322.530244] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 322.570112] Lustre: Skipped 2 previous similar messages [ 323.983826] Lustre: server umount lustre-MDT0001 complete [ 328.207285] Lustre: server umount lustre-OST0000 complete [ 332.385765] Lustre: server umount lustre-OST0001 complete [ 339.781197] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 349.506085] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 393.497823] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 393.771558] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 393.796665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 393.864980] Lustre: lustre-MDT0000: new disk, initializing [ 393.975817] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 394.005321] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 399.005770] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 409.346670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 409.440785] Lustre: 14995:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 409.485709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 409.488263] Lustre: Skipped 1 previous similar message [ 409.539557] Lustre: lustre-MDT0001: new disk, initializing [ 409.633512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 409.669064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 409.680560] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 413.579354] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 422.153563] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 425.755442] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 433.426551] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 433.631400] Lustre: lustre-OST0000: new disk, initializing [ 433.634391] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 433.639250] Lustre: 16741:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 433.723544] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 435.083677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 435.090317] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 435.297768] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 442.147803] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 450.937805] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 457.260087] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 457.502856] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 462.140821] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 462.419274] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 465.895211] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 465.912300] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 465.941102] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 465.944053] Lustre: Skipped 1 previous similar message [ 470.566700] Lustre: server umount lustre-OST0000 complete [ 476.136690] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 476.140597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 476.153426] Lustre: Skipped 1 previous similar message [ 476.156092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 476.168420] LustreError: Skipped 1 previous similar message [ 476.201086] Lustre: Skipped 1 previous similar message [ 480.646834] Lustre: server umount lustre-MDT0000 complete [ 481.265672] LustreError: 16636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 481.298110] LustreError: 16636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 485.026592] LustreError: 14987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335814 with bad export cookie 13467849139632487930 [ 485.027359] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 485.039182] LustreError: 14987:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 485.339560] Lustre: server umount lustre-MDT0001 complete [ 490.441595] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 13:43:38 (1780335818) [ 493.326981] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 495.320807] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 13:43:43 (1780335823) [ 503.129930] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.858501] LustreError: 19004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 503.995683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 508.845408] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 509.410498] LustreError: 19005:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 517.910599] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 521.961638] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 528.040804] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 531.784844] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 538.806350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 545.438586] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 552.400078] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 564.193144] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 564.196555] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 564.198809] LustreError: Skipped 1 previous similar message [ 564.203180] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 564.241586] Lustre: Skipped 2 previous similar messages [ 566.964527] Lustre: server umount lustre-OST0000 complete [ 574.442760] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 585.184204] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 585.387300] Lustre: server umount lustre-MDT0000 complete [ 585.699167] LustreError: 19004:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 585.708328] LustreError: 19004:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 589.710300] LustreError: 18986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780335918 with bad export cookie 13467849139632488861 [ 589.715926] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 589.728021] LustreError: 18986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 590.080377] Lustre: server umount lustre-MDT0001 complete [ 602.895184] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 606.387671] Key type lgssc unregistered [ 606.793135] LNet: 22183:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 606.799804] LNetError: 22183:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 606.834961] LNet: Removed LNI 192.168.202.107@tcp [ 607.921153] Key type .llcrypt unregistered [ 607.923385] Key type ._llcrypt unregistered [ 630.465532] Key type ._llcrypt registered [ 630.468799] Key type .llcrypt registered [ 630.586923] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 632.095610] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 632.129241] alg: No test for adler32 (adler32-zlib) [ 633.205439] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 633.434966] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 635.160190] Key type lgssc registered [ 636.368729] Lustre: Echo OBD driver; http://www.lustre.org/ [ 644.925967] loop: module loaded [ 644.990798] loop0: detected capacity change from 0 to 204800000 [ 645.293231] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 645.311091] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 645.701717] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 652.035052] loop0: detected capacity change from 0 to 204800000 [ 652.205900] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 652.226613] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 652.234836] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 658.171365] loop0: detected capacity change from 0 to 204800000 [ 664.312284] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 664.336464] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 665.740388] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 665.813056] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 665.996385] Lustre: lustre-MDT0000: new disk, initializing [ 666.221523] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 666.255468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 666.371745] Lustre: 24372:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 249 < left 263, rollback = 2 [ 666.394249] Lustre: 24372:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 666.411196] Lustre: 24372:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 2/263/0 [ 666.430245] Lustre: 24372:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 0/0/0, punch: 0/0/0, quota 1/3/2 [ 666.442706] Lustre: 24372:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/5, delete: 0/0/0 [ 666.452810] Lustre: 24372:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 671.266123] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 679.886452] loop1: detected capacity change from 0 to 204800000 [ 685.849353] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 686.349604] Lustre: lustre-OST07c6: new disk, initializing [ 686.356223] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 686.364782] Lustre: Skipped 1 previous similar message [ 686.370249] Lustre: 25335:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 686.502380] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 695.094131] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 695.112495] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 695.120796] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 695.181666] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 701.925472] Lustre: 24337:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 701.933311] Lustre: 24337:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 701.939300] Lustre: 24337:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 701.947296] Lustre: 24337:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 701.954902] Lustre: 24337:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 701.964815] Lustre: 24337:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 720.736811] Lustre: 24338:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 720.750381] Lustre: 24338:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 720.756170] Lustre: 24338:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 720.765654] Lustre: 24338:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 720.771950] Lustre: 24338:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 720.777439] Lustre: 24338:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 726.141796] Lustre: 24338:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 252 < left 278, rollback = 2 [ 726.146195] Lustre: 24338:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 726.149298] Lustre: 24338:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 726.152574] Lustre: 24338:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.162265] Lustre: 24338:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 726.169963] Lustre: 24338:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.183177] Lustre: 24338:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 726.197146] Lustre: 24338:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.204578] Lustre: 24338:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 726.212230] Lustre: 24338:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 726.226512] Lustre: 24338:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 726.236595] Lustre: 24338:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 734.786754] Lustre: 24339:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 734.796375] Lustre: 24339:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 734.801842] Lustre: 24339:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 734.808997] Lustre: 24339:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 734.814195] Lustre: 24339:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 734.819954] Lustre: 24339:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 734.825123] Lustre: 24339:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 734.831909] Lustre: 24339:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 734.836694] Lustre: 24339:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 734.843389] Lustre: 24339:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 734.849603] Lustre: 24339:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 734.857244] Lustre: 24339:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 746.072651] Lustre: 24339:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 278, rollback = 2 [ 746.086670] Lustre: 24339:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 746.096377] Lustre: 24339:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 746.113164] Lustre: 24339:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.120967] Lustre: 24339:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 746.129021] Lustre: 24339:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.138068] Lustre: 24339:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/1 [ 746.154466] Lustre: 24339:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.161949] Lustre: 24339:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 746.178033] Lustre: 24339:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 746.187050] Lustre: 24339:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 746.195055] Lustre: 24339:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 797.670300] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 797.674108] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 797.709123] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 801.967260] Lustre: server umount lustre-OST07c6 complete [ 815.765052] Lustre: server umount lustre-MDT0000 complete [ 836.207973] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 839.206776] Key type lgssc unregistered [ 839.453784] LNet: 27856:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 839.467739] LNetError: 27856:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 839.492245] LNet: Removed LNI 192.168.202.107@tcp [ 840.346230] Key type .llcrypt unregistered [ 840.348063] Key type ._llcrypt unregistered [ 857.862450] Key type ._llcrypt registered [ 857.865147] Key type .llcrypt registered [ 857.954184] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 858.922677] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 859.013864] alg: No test for adler32 (adler32-zlib) [ 860.168250] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 860.410684] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 862.136226] Key type lgssc registered [ 863.733343] Lustre: Echo OBD driver; http://www.lustre.org/ [ 871.314654] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 871.332856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.870361] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 877.567217] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 882.160691] LustreError: 29111:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 882.189114] LustreError: 29111:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 886.605991] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 886.908912] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 890.970607] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 897.135191] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 902.615581] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 912.896173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 913.098344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 917.250857] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 919.875858] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 928.174827] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 952.291244] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 952.295496] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 952.308928] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 956.157757] Lustre: server umount lustre-OST0000 complete [ 962.529426] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 962.552471] Lustre: Skipped 1 previous similar message [ 962.558068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 962.563985] Lustre: Skipped 2 previous similar messages [ 966.509140] Lustre: server umount lustre-MDT0000 complete [ 967.651207] LustreError: 30522:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 967.673885] LustreError: 30522:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 970.319814] LustreError: 29093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336299 with bad export cookie 2038943608362469439 [ 970.323984] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 970.333589] LustreError: 29093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 970.692597] Lustre: server umount lustre-MDT0001 complete [ 982.692772] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 985.764828] Key type lgssc unregistered [ 986.150199] LNet: 32492:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 986.154578] LNetError: 32492:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 986.200337] LNet: Removed LNI 192.168.202.107@tcp [ 987.191854] Key type .llcrypt unregistered [ 987.194225] Key type ._llcrypt unregistered [ 1006.880875] Key type ._llcrypt registered [ 1006.883253] Key type .llcrypt registered [ 1007.020487] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 1008.191727] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1008.401065] alg: No test for adler32 (adler32-zlib) [ 1009.538806] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 1009.774814] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 1011.472274] Key type lgssc registered [ 1012.836733] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1020.672648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1020.698275] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1022.135707] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1027.735206] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1032.680031] LustreError: 33763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1032.704023] LustreError: 33763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1037.801319] LustreError: 33762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1038.608710] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1039.058829] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1044.486625] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1051.135941] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1054.864641] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1062.952165] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1063.463874] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.648722] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 1070.094470] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1078.136838] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1090.017929] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1090.023660] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1090.038333] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1093.606583] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1093.613030] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1093.614711] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1093.631310] Lustre: Skipped 1 previous similar message [ 1095.761555] Lustre: server umount lustre-OST0000 complete [ 1100.771076] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1100.780508] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1100.793190] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1103.858393] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1103.881538] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1106.671951] Lustre: server umount lustre-MDT0000 complete [ 1108.964801] LustreError: 35910:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1108.989643] LustreError: 35910:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1112.165045] LustreError: 33741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336441 with bad export cookie 5981120465443878760 [ 1112.176297] LustreError: 33741:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1112.179871] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1112.825712] Lustre: server umount lustre-MDT0001 complete [ 1126.178955] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 1128.921691] Key type lgssc unregistered [ 1129.168579] LNet: 36944:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1129.181462] LNetError: 36944:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1129.193342] LNet: Removed LNI 192.168.202.107@tcp [ 1129.882347] Key type .llcrypt unregistered [ 1129.884325] Key type ._llcrypt unregistered [ 1157.930104] Key type ._llcrypt registered [ 1157.931870] Key type .llcrypt registered [ 1158.017189] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 1159.153944] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1159.358765] alg: No test for adler32 (adler32-zlib) [ 1160.580949] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 1160.859760] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 1162.552175] Key type lgssc registered [ 1164.114609] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1173.163721] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1173.186555] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1174.674964] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1180.085733] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1184.233403] LustreError: 38346:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1184.257522] LustreError: 38346:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1189.351864] LustreError: 38347:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1190.390388] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1190.729539] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1195.713447] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1202.690687] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1207.243652] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1215.134768] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1215.399948] LustreError: 39943:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1215.557466] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1219.664861] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 1222.771742] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1229.801287] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1238.043219] LustreError: 38341:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1238.200257] Lustre: server umount lustre-MDT0000 complete [ 1240.035989] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1240.070267] Lustre: Skipped 2 previous similar messages [ 1242.110677] LustreError: 39967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336571 with bad export cookie 4903924803210013421 [ 1242.131069] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1242.132849] LustreError: 39967:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1243.111580] LustreError: 38342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1243.121082] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.7@tcp (stopping) [ 1243.127089] LustreError: 38342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1245.155885] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1245.182685] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1247.201916] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1248.763026] Lustre: server umount lustre-MDT0001 complete [ 1270.649921] Lustre: server umount lustre-OST0000 complete [ 1291.259689] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 1294.173900] Key type lgssc unregistered [ 1294.591414] LNet: 41857:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1294.598275] LNetError: 41857:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1294.627310] LNet: Removed LNI 192.168.202.107@tcp [ 1295.542805] Key type .llcrypt unregistered [ 1295.548431] Key type ._llcrypt unregistered [ 1321.518788] Key type ._llcrypt registered [ 1321.523757] Key type .llcrypt registered [ 1321.646201] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 1322.870950] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1322.959677] alg: No test for adler32 (adler32-zlib) [ 1324.008980] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 1324.204195] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 1325.928858] Key type lgssc registered [ 1327.324632] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1335.679466] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1335.706465] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1337.410028] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1342.463612] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1347.565860] LustreError: 43261:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1347.590694] LustreError: 43261:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1352.359311] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.688612] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 1352.947402] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1357.931924] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1364.334619] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1369.527774] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1375.598277] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1376.074917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1382.200877] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 1382.245383] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1388.027483] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1394.721500] Lustre: lustre-OST0000: Not available for connect from 192.168.202.7@tcp (stopping) [ 1397.730410] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1397.744108] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1397.753217] Lustre: Skipped 1 previous similar message [ 1397.759076] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1397.772032] Lustre: Skipped 1 previous similar message [ 1399.776061] Lustre: lustre-OST0000: Not available for connect from 192.168.202.7@tcp (stopping) [ 1400.762114] Lustre: server umount lustre-OST0000 complete [ 1419.754368] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1419.764579] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1419.783590] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1422.319738] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1425.749893] Lustre: server umount lustre-MDT0000 complete [ 1427.428605] LustreError: 43257:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1427.472382] LustreError: 43257:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1430.135260] LustreError: 43243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336759 with bad export cookie 3161018921199168502 [ 1430.136722] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1430.141400] LustreError: 43243:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1436.640808] Lustre: server umount lustre-MDT0001 complete [ 1454.759700] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 1458.646717] Key type lgssc unregistered [ 1458.980503] LNet: 46618:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1458.987882] LNetError: 46618:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1459.013834] LNet: Removed LNI 192.168.202.107@tcp [ 1460.120800] Key type .llcrypt unregistered [ 1460.123468] Key type ._llcrypt unregistered [ 1491.821741] Key type ._llcrypt registered [ 1491.824740] Key type .llcrypt registered [ 1491.930699] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 1493.361839] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1493.403208] alg: No test for adler32 (adler32-zlib) [ 1494.412439] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 1494.623451] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 1496.310480] Key type lgssc registered [ 1497.704251] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1506.402656] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1506.433364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.968070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1512.905595] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1517.553407] LustreError: 48019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1517.585061] LustreError: 48019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1522.269208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1522.592110] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1527.031936] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1532.822489] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1536.428765] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1544.018464] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1544.352624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1551.187831] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1552.567455] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 1557.477275] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1563.721477] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1567.912310] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 1580.381850] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-0114h01m47s [ 1582.656872] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 1585.655843] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.7@tcp (stopping) [ 1586.730893] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.7@tcp (stopping) [ 1588.707860] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1588.712480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1588.759111] Lustre: Skipped 2 previous similar messages [ 1588.761601] Lustre: Skipped 1 previous similar message [ 1591.776418] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.7@tcp (stopping) [ 1591.791089] Lustre: Skipped 4 previous similar messages [ 1592.178913] Lustre: server umount lustre-MDT0000 complete [ 1595.884671] LustreError: 48020:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1595.930091] LustreError: 48020:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1597.137860] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 1597.919695] LustreError: 48014:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1602.033577] LustreError: 48019:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1602.053818] LustreError: 48019:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1605.391776] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1605.494218] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1605.763685] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1605.844971] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 1606.785290] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 1610.475758] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1614.140799] Lustre: DEBUG MARKER: Wait for df (36090) ... [ 1616.305848] Lustre: DEBUG MARKER: done [ 1620.969552] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1620.978161] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1620.986603] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1622.497453] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1632.748047] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1632.766965] Lustre: Skipped 5 previous similar messages [ 1635.296771] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1635.602510] Lustre: server umount lustre-OST0000 complete [ 1642.984974] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1642.991675] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1643.014779] Lustre: Skipped 1 previous similar message [ 1643.021529] LustreError: 47331:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting [ 1646.075042] Lustre: server umount lustre-MDT0000 complete [ 1650.150217] LustreError: 50595:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1650.166165] LustreError: 50595:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1650.779904] LustreError: 47999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780336979 with bad export cookie 5705748246094441866 [ 1650.786704] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1650.793752] LustreError: 47999:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1651.328419] Lustre: server umount lustre-MDT0001 complete [ 1663.167144] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 1666.669877] Key type lgssc unregistered [ 1666.967461] LNet: 52590:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1666.981900] LNetError: 52590:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1668.008509] LNet: Removed LNI 192.168.202.107@tcp [ 1669.231085] Key type .llcrypt unregistered [ 1669.244046] Key type ._llcrypt unregistered [ 1713.856046] Key type ._llcrypt registered [ 1713.857686] Key type .llcrypt registered [ 1714.097602] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 1715.693695] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1715.864902] alg: No test for adler32 (adler32-zlib) [ 1717.328421] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 1717.652113] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 1719.392503] Key type lgssc registered [ 1721.520402] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1732.700727] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1732.737723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1734.105569] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1734.129528] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1734.282596] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1737.827646] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1745.957094] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1746.119311] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1746.171053] Lustre: 54784:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1746.602675] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1752.261482] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1760.338604] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1764.743940] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1773.558928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1773.820492] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1774.291511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1780.552277] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 1782.597561] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1790.040440] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1797.621546] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-0114h05m25s [ 1799.322433] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1806.613976] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1809.071146] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1809.108251] LustreError: 56979:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1829.422106] Lustre: lustre-MDT0000: Client 9d214ea7-bfee-4a6c-9a7a-dac3d9442953 (at 192.168.202.7@tcp) reconnecting [ 1849.144112] LustreError: 56979:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1851.754388] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1854.273404] Lustre: DEBUG MARKER: done [ 1860.565513] Lustre: server umount lustre-OST0000 complete [ 1867.749134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1867.754180] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1867.766364] Lustre: Skipped 1 previous similar message [ 1867.783839] Lustre: Skipped 1 previous similar message [ 1868.833706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1868.841910] Lustre: Skipped 1 previous similar message [ 1874.915289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1874.926526] Lustre: Skipped 1 previous similar message [ 1876.010205] Lustre: server umount lustre-MDT0000 complete [ 1879.012038] Lustre: 54106:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337191/real 1780337191] req@000000001d996332 x1866818698403840/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1780337207 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1879.046180] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1879.946571] LustreError: 54777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337208 with bad export cookie 7384826423578740637 [ 1879.950190] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1879.962021] LustreError: 54777:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1880.194963] Lustre: server umount lustre-MDT0001 complete [ 1890.889288] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 1893.168161] Key type lgssc unregistered [ 1893.432412] LNet: 58758:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1893.439024] LNetError: 58758:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1893.450970] LNet: Removed LNI 192.168.202.107@tcp [ 1894.135314] Key type .llcrypt unregistered [ 1894.137379] Key type ._llcrypt unregistered [ 1940.831158] Key type ._llcrypt registered [ 1940.836133] Key type .llcrypt registered [ 1940.937807] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 1942.059443] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1942.260688] alg: No test for adler32 (adler32-zlib) [ 1943.376672] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 1943.672948] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 1945.376241] Key type lgssc registered [ 1946.591597] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1949.707606] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1949.732256] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1951.040644] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1951.065591] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1951.184618] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1952.155535] Lustre: Failing over lustre-MDT0000 [ 1952.524038] Lustre: server umount lustre-MDT0000 complete [ 1959.878723] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 14:08:07 (1780337287) [ 1966.682945] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1967.228575] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1971.997425] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1981.674033] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1981.762738] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1981.786854] Lustre: 61538:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1982.106802] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1986.792443] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 1993.586703] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1998.487954] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2006.046266] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2006.251210] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 2006.534306] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2013.414753] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2016.400981] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 2019.602502] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2026.005675] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 2037.220330] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2037.223489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2037.241951] Lustre: Skipped 1 previous similar message [ 2037.262026] Lustre: Skipped 2 previous similar messages [ 2040.145940] Lustre: server umount lustre-MDT0000 complete [ 2044.445623] LustreError: 61531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337373 with bad export cookie 11228477888202021852 [ 2044.449105] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2044.465849] LustreError: 61531:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2044.812731] Lustre: server umount lustre-MDT0001 complete [ 2048.548301] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 2053.572605] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2063.081524] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2063.840462] Lustre: 60463:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337376/real 1780337376] req@0000000007c6dc92 x1866818935365888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780337392 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2063.876785] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2063.911175] Lustre: Skipped 1 previous similar message [ 2065.890823] Lustre: 60464:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337378/real 1780337378] req@0000000072fbb379 x1866818935366016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780337394 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2068.960501] Lustre: 60462:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337381/real 1780337381] req@000000001a3a5fa7 x1866818935366144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780337397 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2070.052293] LustreError: 60460:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000009ef5659e x1866818935366912/t0(0) o250->MGC192.168.202.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2070.379387] LustreError: 63169:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2070.400081] LustreError: 63169:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2070.471078] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2070.580319] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 2071.456182] Lustre: 60461:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337383/real 1780337383] req@00000000bf762901 x1866818935366400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780337399 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2072.489337] LustreError: 63170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.514422] LustreError: 64977:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2073.552196] LustreError: 64977:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2075.124726] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2078.695806] LustreError: 64978:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2078.725432] LustreError: 64978:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2083.825813] LustreError: 63170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2083.842163] LustreError: 63170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2085.762248] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2086.148449] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2090.940537] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2091.500683] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2092.597980] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2097.681946] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2102.082769] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2111.969180] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2111.996738] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2112.031509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2116.067961] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2116.082361] Lustre: Skipped 5 previous similar messages [ 2116.696920] Lustre: server umount lustre-MDT0000 complete [ 2121.200933] LustreError: 66291:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2121.242347] LustreError: 66291:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2121.350797] LustreError: 61530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337450 with bad export cookie 11228477888202024127 [ 2121.355283] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2121.363770] LustreError: 61530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2122.065718] Lustre: server umount lustre-MDT0001 complete [ 2129.923897] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2141.601810] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2142.694594] Lustre: 60464:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337455/real 1780337455] req@00000000f88b1536 x1866818935396608/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780337471 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2142.740623] Lustre: 60464:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2142.756227] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2142.776096] Lustre: Skipped 2 previous similar messages [ 2147.542177] LustreError: 63170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2147.723504] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2147.833678] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 2151.803582] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2152.868302] Lustre: 60462:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780337465/real 1780337465] req@0000000035eae540 x1866818935396992/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780337481 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2152.900710] Lustre: 60462:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2155.527825] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2168.033584] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2173.443251] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 2174.392536] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2181.990084] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2186.614513] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2193.890477] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2193.894861] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2193.926775] Lustre: Skipped 2 previous similar messages [ 2193.946524] Lustre: Skipped 2 previous similar messages [ 2199.012309] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2199.018528] Lustre: Skipped 2 previous similar messages [ 2199.904337] Lustre: server umount lustre-MDT0000 complete [ 2203.840295] LustreError: 61530:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337532 with bad export cookie 11228477888202025506 [ 2203.851871] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2204.211277] Lustre: server umount lustre-MDT0001 complete [ 2211.715667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 2215.852854] Lustre: DEBUG MARKER: files compared the same [ 2227.555520] Lustre: server umount lustre-OST0000 complete [ 2246.753628] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 2249.438848] Key type lgssc unregistered [ 2249.820892] LNet: 70473:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2249.839511] LNetError: 70473:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2249.875352] LNet: Removed LNI 192.168.202.107@tcp [ 2250.627982] Key type .llcrypt unregistered [ 2250.631695] Key type ._llcrypt unregistered [ 2273.921874] Key type ._llcrypt registered [ 2273.924324] Key type .llcrypt registered [ 2274.015748] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2275.015333] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2275.111132] alg: No test for adler32 (adler32-zlib) [ 2276.402734] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 2276.625949] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 2278.296336] Key type lgssc registered [ 2279.647582] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2287.654572] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2287.677114] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2289.172690] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2293.402533] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug +malloc all [ 2299.366212] LustreError: 71872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2299.378819] LustreError: 71872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2302.880115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2303.266116] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2308.339550] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug +malloc all [ 2315.211224] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2319.748943] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2328.462386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2329.072336] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2334.201841] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 2336.232631] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug +malloc all [ 2342.943940] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2354.658158] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2354.660822] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2354.698106] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2354.709141] Lustre: Skipped 1 previous similar message [ 2359.218516] Lustre: server umount lustre-OST0000 complete [ 2364.897313] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2364.899638] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2364.918323] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2364.927575] Lustre: Skipped 2 previous similar messages [ 2370.016951] LustreError: 72839:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2370.040742] LustreError: 72839:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2371.677791] Lustre: server umount lustre-MDT0000 complete [ 2374.122309] LustreError: 71871:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2374.143645] LustreError: 71871:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2375.718920] LustreError: 71854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337704 with bad export cookie 13133193721285858653 [ 2375.726201] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2375.735308] LustreError: 71854:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2376.007623] Lustre: server umount lustre-MDT0001 complete [ 2386.892308] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 2389.998183] Key type lgssc unregistered [ 2390.330229] LNet: 75054:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2390.340048] LNetError: 75054:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2391.399523] LNet: Removed LNI 192.168.202.107@tcp [ 2392.104988] Key type .llcrypt unregistered [ 2392.107454] Key type ._llcrypt unregistered [ 2416.685622] Key type ._llcrypt registered [ 2416.688478] Key type .llcrypt registered [ 2416.786518] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2417.870959] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2418.094503] alg: No test for adler32 (adler32-zlib) [ 2419.078263] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 2419.353378] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 2421.048195] Key type lgssc registered [ 2422.412337] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2430.429498] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2430.449033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2445.984364] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 2451.104583] LustreError: 76427:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.107@tcp: failed processing log, type 4: rc = -110 [ 2477.861390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2485.994435] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2492.910523] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2501.776566] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2501.875784] Lustre: *** cfs_fail_loc=706, val=0*** [ 2502.391049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2502.466816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 2507.446740] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2507.749082] LustreError: 76452:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2507.770625] LustreError: 76452:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2512.873325] LustreError: 76453:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2512.896906] LustreError: 76453:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2516.762962] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2517.159674] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2521.898681] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2528.558845] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2532.692380] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2537.975866] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2537.976547] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2537.986922] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2541.410586] Lustre: server umount lustre-OST0000 complete [ 2547.683803] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2547.697282] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2547.714395] Lustre: Skipped 1 previous similar message [ 2547.726492] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2547.735476] Lustre: Skipped 1 previous similar message [ 2552.331970] Lustre: server umount lustre-MDT0000 complete [ 2553.314954] LustreError: 77405:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2553.339546] LustreError: 77405:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2555.974736] LustreError: 78125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780337884 with bad export cookie 15262437489306925476 [ 2555.978116] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2555.980877] LustreError: 78125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2556.565968] Lustre: server umount lustre-MDT0001 complete [ 2570.933189] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 2573.584375] Key type lgssc unregistered [ 2573.875744] LNet: 79686:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2573.886522] LNetError: 79686:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2574.952073] LNet: Removed LNI 192.168.202.107@tcp [ 2576.260793] Key type .llcrypt unregistered [ 2576.263867] Key type ._llcrypt unregistered [ 2607.325327] Key type ._llcrypt registered [ 2607.329658] Key type .llcrypt registered [ 2607.486148] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2608.702065] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2608.831315] alg: No test for adler32 (adler32-zlib) [ 2609.820819] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 2610.016526] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 2611.712525] Key type lgssc registered [ 2613.213784] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2620.696163] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2620.715139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2626.428600] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2634.022284] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2634.453870] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2638.303604] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2639.851408] LustreError: 81759:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2639.875046] LustreError: 81759:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2644.966082] LustreError: 81763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2645.000568] LustreError: 81763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2649.431410] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2649.739787] LustreError: 81759:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2649.761132] LustreError: 81759:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2649.780655] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2655.207278] LustreError: 82489:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2655.243089] LustreError: 82489:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2657.247591] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2659.304945] LustreError: 81762:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2659.340274] LustreError: 81762:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2667.988373] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2667.998468] Lustre: MGS: Client 093a26ed-ae66-4835-8f53-bf67fcf35dba (at 0@lo) reconnecting [ 2668.028857] LustreError: 81773:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2668.463779] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2668.579804] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 2672.930403] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2688.993174] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2688.996433] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2689.009078] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2689.021357] Lustre: Skipped 1 previous similar message [ 2694.117959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2694.126678] Lustre: Skipped 1 previous similar message [ 2694.319244] Lustre: server umount lustre-OST0000 complete [ 2704.353414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2704.355607] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2704.378030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2704.385176] Lustre: Skipped 1 previous similar message [ 2704.418877] Lustre: Skipped 2 previous similar messages [ 2705.818762] Lustre: server umount lustre-MDT0000 complete [ 2709.478560] LustreError: 81763:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2709.509413] LustreError: 81763:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 2710.057698] Lustre: server umount lustre-MDT0001 complete [ 2714.975410] Lustre: server umount MGS complete [ 2731.199243] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 14:20:58 (1780338058) [ 2769.333223] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 2784.361369] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 2827.351438] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2833.723356] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2846.640985] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2846.963822] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 2847.076187] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2847.123161] Lustre: lustre-MDT0001: new disk, initializing [ 2847.217318] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2847.238272] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2847.245156] Lustre: 88875:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 2847.252782] LustreError: 88875:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 2847.269817] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2847.282462] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2847.769106] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2847.775210] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) Skipped 3839 previous similar messages [ 2847.779611] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2847.783035] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 3839 previous similar messages [ 2848.769095] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2848.775350] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) Skipped 7052 previous similar messages [ 2848.779424] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2848.784990] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 7052 previous similar messages [ 2850.771219] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2850.788427] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) Skipped 9461 previous similar messages [ 2850.799634] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2850.805401] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 9461 previous similar messages [ 2852.639056] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2854.771045] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2854.785811] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) Skipped 15590 previous similar messages [ 2854.799095] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2854.812578] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 15632 previous similar messages [ 2862.771112] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2862.785461] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) Skipped 38606 previous similar messages [ 2862.805798] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2862.824267] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 38584 previous similar messages [ 2866.518215] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2866.742156] Lustre: lustre-OST0000: new disk, initializing [ 2866.745464] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2866.750214] Lustre: 89628:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2866.836848] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2873.302373] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2874.914411] Lustre: 89985:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@0000000068e75a9a x1866819633721984/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780338219 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2878.774297] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) srv-lustre-MDT0001: lookup 0x200000002, but not connects to MDT0 yet: rc = -5. [ 2878.792943] LustreError: 88877:0:(fld_handler.c:275:fld_server_lookup()) Skipped 129382 previous similar messages [ 2878.809962] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) lustre-MDT0001: Trash dir setup failed: rc = -5 [ 2878.819648] LustreError: 88877:0:(mdt_handler.c:8457:mdt_trash_setup_thread()) Skipped 129362 previous similar messages [ 2882.851110] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2891.389499] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2891.395640] Lustre: MGS: Client dd37e73d-fd70-45e6-aedf-30809c68e84c (at 0@lo) reconnecting [ 2891.445677] Lustre: 88161:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2891.462777] LustreError: 88861:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 2891.482595] LustreError: 88861:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 2891.531725] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2891.686413] Lustre: lustre-MDT0000: new disk, initializing [ 2891.888421] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2891.951128] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2891.993168] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2893.042204] Lustre: 90585:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000c5e5bd9f x1866819633729024/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1780338238 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 2898.051434] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 2898.224855] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 2901.766595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2901.780928] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2902.322225] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2902.372357] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 2927.626217] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2927.628758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2927.635748] Lustre: Skipped 1 previous similar message [ 2932.712903] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2932.719514] Lustre: Skipped 2 previous similar messages [ 2939.362034] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2939.652288] Lustre: server umount lustre-OST0000 complete [ 2948.068189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2948.073173] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2948.077574] Lustre: Skipped 1 previous similar message [ 2948.089149] Lustre: Skipped 3 previous similar messages [ 2950.581342] Lustre: server umount lustre-MDT0000 complete [ 2953.191657] LustreError: 88845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2953.208966] LustreError: 88845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2960.556846] Lustre: server umount lustre-MDT0001 complete [ 2964.201068] Lustre: server umount MGS complete [ 2979.156529] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 14:25:06 (1780338306) [ 2998.388611] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 3001.823445] Key type lgssc unregistered [ 3002.181672] LNet: 93049:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3002.201507] LNetError: 93049:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3002.219744] LNet: Removed LNI 192.168.202.107@tcp [ 3003.058777] Key type .llcrypt unregistered [ 3003.061142] Key type ._llcrypt unregistered [ 3021.307292] Key type ._llcrypt registered [ 3021.310767] Key type .llcrypt registered [ 3021.467759] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3022.946201] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3023.020717] alg: No test for adler32 (adler32-zlib) [ 3024.106167] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 3024.332728] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 3026.072230] Key type lgssc registered [ 3027.496758] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3036.425411] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3036.449268] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3038.173985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3043.761123] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3048.427707] LustreError: 94255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3048.443636] LustreError: 94255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3053.553753] LustreError: 94256:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3053.821280] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.201799] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3060.111757] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3066.920700] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3070.338368] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3077.703864] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3078.015719] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3083.140157] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 3084.812136] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3091.213575] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3100.643023] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3100.661579] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3100.690763] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3101.355350] Lustre: server umount lustre-OST0000 complete [ 3108.839281] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3108.845332] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3108.866410] Lustre: Skipped 1 previous similar message [ 3113.952910] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3113.964710] Lustre: Skipped 2 previous similar messages [ 3116.071583] Lustre: server umount lustre-MDT0000 complete [ 3118.050763] LustreError: 94255:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3118.085360] LustreError: 94255:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3119.328089] LustreError: 96410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338448 with bad export cookie 9384913093944810896 [ 3119.340183] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3119.346844] LustreError: 96410:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3119.622136] Lustre: server umount lustre-MDT0001 complete [ 3131.831672] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 3134.228950] Key type lgssc unregistered [ 3134.440682] LNet: 97432:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3134.447100] LNetError: 97432:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3134.462107] LNet: Removed LNI 192.168.202.107@tcp [ 3135.262265] Key type .llcrypt unregistered [ 3135.265168] Key type ._llcrypt unregistered [ 3151.407141] Key type ._llcrypt registered [ 3151.412963] Key type .llcrypt registered [ 3151.530936] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3152.269702] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3152.370349] alg: No test for adler32 (adler32-zlib) [ 3153.516703] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 3153.852491] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 3155.592315] Key type lgssc registered [ 3157.353486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3164.522279] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3164.529590] LustreError: 98492:0:(tgt_mount.c:2525:server_fill_super()) cfs_race id 716 sleeping [ 3164.550086] LustreError: 98502:0:(tgt_mount.c:2525:server_fill_super()) cfs_fail_race id 716 waking [ 3164.553707] LustreError: 98492:0:(tgt_mount.c:2525:server_fill_super()) cfs_fail_race id 716 awake: rc=4980 [ 3164.561892] LustreError: 98492:0:(tgt_mount.c:2537:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 3164.566788] LustreError: 98492:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3164.576723] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3165.940186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3173.426331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3173.670909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3178.268731] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3184.252944] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing lsmod [ 3186.900059] LustreError: 99675:0:(tgt_mount.c:2525:server_fill_super()) cfs_race id 716 sleeping [ 3186.906386] LustreError: 99676:0:(tgt_mount.c:2525:server_fill_super()) cfs_fail_race id 716 waking [ 3186.913412] LustreError: 99675:0:(tgt_mount.c:2525:server_fill_super()) cfs_fail_race id 716 awake: rc=4995 [ 3186.922788] LustreError: 99675:0:(tgt_mount.c:2537:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 3186.925350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3186.932652] LustreError: 99675:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 3187.349248] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3190.491284] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3190.497870] LustreError: 98558:0:(osp_precreate.c:970:osp_precreate_cleanup_orphans()) lustre-OST0000-osc-MDT0000: cannot cleanup orphans: rc = -5 [ 3194.337692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3194.350986] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3194.374645] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3195.878273] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3195.879210] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3195.905326] Lustre: Skipped 1 previous similar message [ 3195.908954] Lustre: Skipped 1 previous similar message [ 3197.039574] Lustre: server umount lustre-MDT0000 complete [ 3199.012671] LustreError: 98538:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3199.032591] LustreError: 98538:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 3201.174615] LustreError: 98514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338530 with bad export cookie 7924296843194762591 [ 3201.181945] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3201.190110] LustreError: 98514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3203.169458] LustreError: 98970:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000006f2ef89b x1866820204123648/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3203.396878] Lustre: server umount lustre-MDT0001 complete [ 3217.953488] Lustre: server umount lustre-OST0000 complete [ 3227.158135] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3227.402186] LustreError: 100740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3227.462367] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3232.702825] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3232.748414] LustreError: 100741:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3237.872835] LustreError: 100740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3242.131742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3242.391187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3247.481675] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3254.330686] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3258.656635] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3265.774391] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3266.311850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3272.441649] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 3272.724400] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3278.619379] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3286.888318] Lustre: server umount lustre-OST0000 complete [ 3293.154135] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.154630] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3293.163617] Lustre: Skipped 1 previous similar message [ 3293.174465] Lustre: Skipped 2 previous similar messages [ 3298.274240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3298.287699] Lustre: Skipped 1 previous similar message [ 3300.460533] Lustre: server umount lustre-MDT0000 complete [ 3303.393563] LustreError: 100737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.408828] LustreError: 100737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3304.137906] LustreError: 100721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338633 with bad export cookie 7924296843194763522 [ 3304.150987] LustreError: 100721:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3304.153153] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3314.926567] Lustre: server umount lustre-MDT0001 complete [ 3328.238615] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 3331.633424] Key type lgssc unregistered [ 3331.958758] LNet: 103919:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3331.964984] LNetError: 103919:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3331.977410] LNet: Removed LNI 192.168.202.107@tcp [ 3332.945769] Key type .llcrypt unregistered [ 3332.948949] Key type ._llcrypt unregistered [ 3358.828264] Key type ._llcrypt registered [ 3358.830717] Key type .llcrypt registered [ 3358.952492] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3360.515768] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3360.562656] alg: No test for adler32 (adler32-zlib) [ 3361.658533] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 3362.118672] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 3363.832701] Key type lgssc registered [ 3365.072430] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3374.047441] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3374.088449] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3375.812234] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3381.159293] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3385.321479] LustreError: 105321:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3385.350129] LustreError: 105321:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3390.475719] LustreError: 105320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3392.185208] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3392.474308] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3397.226475] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3403.310568] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3407.672686] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3414.528224] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3414.823454] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3421.017193] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 3421.724707] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3428.998471] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3434.034946] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 3437.104036] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 3441.635953] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3441.638439] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3441.657635] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3446.760483] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3446.771447] Lustre: Skipped 2 previous similar messages [ 3447.620367] Lustre: server umount lustre-OST0000 complete [ 3451.873576] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3451.877414] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3451.878686] Lustre: Skipped 2 previous similar messages [ 3456.994293] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3457.007817] Lustre: Skipped 2 previous similar messages [ 3462.116774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3462.127546] Lustre: Skipped 1 previous similar message [ 3466.214171] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3466.417379] Lustre: server umount lustre-MDT0000 complete [ 3467.233924] LustreError: 105320:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3467.256464] LustreError: 105320:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3470.630442] LustreError: 107514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338799 with bad export cookie 3922492335686251593 [ 3470.634757] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3470.642218] LustreError: 107514:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3471.039703] Lustre: server umount lustre-MDT0001 complete [ 3483.096198] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 3485.873335] Key type lgssc unregistered [ 3486.077293] LNet: 108605:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3486.086648] LNetError: 108605:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3486.108408] LNet: Removed LNI 192.168.202.107@tcp [ 3487.189963] Key type .llcrypt unregistered [ 3487.193910] Key type ._llcrypt unregistered [ 3504.419798] Key type ._llcrypt registered [ 3504.422144] Key type .llcrypt registered [ 3504.509313] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3505.308812] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3505.326796] alg: No test for adler32 (adler32-zlib) [ 3506.331818] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 3506.514456] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 3508.192185] Key type lgssc registered [ 3509.337844] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3516.244908] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3516.267246] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3517.711955] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3523.492797] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3527.137761] LustreError: 109810:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3527.147849] LustreError: 109810:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3532.268671] LustreError: 109809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3533.117471] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3533.735647] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3539.354330] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3546.531259] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3551.471475] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3558.033662] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3558.326732] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3562.430459] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 3565.231427] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3572.519120] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3582.946871] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3582.969824] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3582.978038] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3583.003038] Lustre: Skipped 1 previous similar message [ 3588.074164] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3588.087315] Lustre: Skipped 1 previous similar message [ 3593.202829] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3593.212461] Lustre: Skipped 1 previous similar message [ 3595.232102] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3595.390380] Lustre: server umount lustre-OST0000 complete [ 3600.356221] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3600.366300] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3600.393212] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3600.399894] Lustre: Skipped 1 previous similar message [ 3603.428594] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3606.062039] Lustre: server umount lustre-MDT0000 complete [ 3608.545130] LustreError: 109809:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3608.559520] LustreError: 109809:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3610.241707] LustreError: 109790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780338939 with bad export cookie 3016262253550204049 [ 3610.253826] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3610.257621] LustreError: 109790:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3610.550699] Lustre: server umount lustre-MDT0001 complete [ 3622.851499] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 3625.692410] Key type lgssc unregistered [ 3626.195893] LNet: 112989:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3626.209780] LNetError: 112989:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3626.230490] LNet: Removed LNI 192.168.202.107@tcp [ 3627.546364] Key type .llcrypt unregistered [ 3627.547993] Key type ._llcrypt unregistered [ 3657.817493] Key type ._llcrypt registered [ 3657.825699] Key type .llcrypt registered [ 3657.988160] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3659.551023] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3659.623939] alg: No test for adler32 (adler32-zlib) [ 3660.663837] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 3660.884064] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 3662.568157] Key type lgssc registered [ 3664.034715] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3671.868596] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3671.882570] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3673.341246] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3678.323947] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3683.815228] LustreError: 114393:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3683.857225] LustreError: 114393:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3687.254750] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3687.828244] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3695.690547] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3704.129142] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3707.798779] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3716.411632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3716.612606] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3718.715503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3723.015499] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3729.266337] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3739.155992] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 3743.736553] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 3743.743260] Lustre: Skipped 2 previous similar messages [ 3749.227071] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 3749.234941] Lustre: Skipped 2 previous similar messages [ 3749.483091] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 3749.574986] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3755.286728] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.202.7@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 3755.301757] Lustre: Skipped 2 previous similar messages [ 3757.070703] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.202.7@tcp 192.168.0.[2,10]@tcp [ 3763.001995] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.202.7/24@tcp in log lustre-MDT0000 [ 3763.014034] Lustre: Skipped 2 previous similar messages [ 3765.142149] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.7/24@tcp [ 3765.154056] Lustre: Skipped 1 previous similar message [ 3773.927745] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3773.932772] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3773.953804] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3773.966852] Lustre: Skipped 1 previous similar message [ 3779.044163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3779.956511] Lustre: server umount lustre-OST0000 complete [ 3789.283448] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3789.286367] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3789.302985] Lustre: Skipped 2 previous similar messages [ 3789.326173] Lustre: Skipped 2 previous similar messages [ 3792.214603] Lustre: server umount lustre-MDT0000 complete [ 3794.411158] LustreError: 116634:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3794.449394] LustreError: 116634:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3796.892975] LustreError: 116684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339125 with bad export cookie 10118073282479056153 [ 3796.907114] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3796.912844] LustreError: 116684:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3797.368280] Lustre: server umount lustre-MDT0001 complete [ 3809.793964] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 3814.394098] Key type lgssc unregistered [ 3814.848284] LNet: 118683:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3814.862297] LNetError: 118683:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3814.881893] LNet: Removed LNI 192.168.202.107@tcp [ 3815.861538] Key type .llcrypt unregistered [ 3815.866447] Key type ._llcrypt unregistered [ 3847.093735] Key type ._llcrypt registered [ 3847.096313] Key type .llcrypt registered [ 3847.219861] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 3849.193604] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3849.208862] alg: No test for adler32 (adler32-zlib) [ 3850.308471] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 3850.525644] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 3852.168323] Key type lgssc registered [ 3853.661922] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3863.100759] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3863.111487] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3864.844783] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 3864.874732] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3864.891274] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.202.7/24@tcp [ 3871.175509] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3874.279887] LustreError: 120232:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3874.289673] LustreError: 120232:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3879.406306] LustreError: 120233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3882.237849] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3882.567338] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 3882.597273] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.202.7/24@tcp [ 3882.706883] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3888.689452] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3896.181932] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3902.153608] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3911.900129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3912.547785] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3920.609921] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 3928.971467] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3934.104816] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 3948.513965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3948.521204] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3948.546680] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3949.539138] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3949.559663] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3949.574740] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3949.607929] Lustre: Skipped 1 previous similar message [ 3952.539335] Lustre: server umount lustre-OST0000 complete [ 3959.266545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3959.276768] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3959.296171] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3963.692331] Lustre: server umount lustre-MDT0000 complete [ 3964.899833] LustreError: 120233:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3964.920460] LustreError: 120233:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3968.918800] LustreError: 120212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339297 with bad export cookie 3279316199209176493 [ 3968.925537] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3968.927161] LustreError: 120212:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3969.477362] Lustre: server umount lustre-MDT0001 complete [ 3981.672589] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 3984.806631] Key type lgssc unregistered [ 3985.132745] LNet: 123486:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3985.142566] LNetError: 123486:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3985.164499] LNet: Removed LNI 192.168.202.107@tcp [ 3986.123136] Key type .llcrypt unregistered [ 3986.125719] Key type ._llcrypt unregistered [ 4018.136892] Key type ._llcrypt registered [ 4018.139607] Key type .llcrypt registered [ 4018.247973] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 4034.124969] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 4035.791919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4035.831152] alg: No test for adler32 (adler32-zlib) [ 4037.150214] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 4037.496132] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 4039.320184] Key type lgssc registered [ 4040.849807] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4082.291278] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4082.334868] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4083.819635] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4083.905328] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4084.033966] Lustre: lustre-MDT0000: new disk, initializing [ 4084.190295] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4084.210799] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4090.444387] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4103.730924] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4103.879206] Lustre: 126858:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4103.940505] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4103.948530] Lustre: Skipped 1 previous similar message [ 4104.034395] Lustre: lustre-MDT0001: new disk, initializing [ 4104.194671] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4104.243792] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4104.260148] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4109.770087] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4119.461824] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4124.306851] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4133.571623] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4133.916554] Lustre: lustre-OST0000: new disk, initializing [ 4133.922860] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4133.929369] Lustre: 128601:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4134.049692] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4141.099713] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4141.108978] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4141.182821] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4141.577269] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4151.025605] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4157.284918] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4157.732870] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4163.122097] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4163.417564] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4166.631855] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4166.649387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4166.659786] Lustre: Skipped 1 previous similar message [ 4171.760139] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4171.774556] Lustre: Skipped 1 previous similar message [ 4172.421990] Lustre: server umount lustre-OST0000 complete [ 4176.866701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4176.877188] Lustre: Skipped 1 previous similar message [ 4176.880691] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4181.987927] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4182.002458] Lustre: Skipped 2 previous similar messages [ 4183.306782] Lustre: server umount lustre-MDT0000 complete [ 4187.652441] LustreError: 126850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339516 with bad export cookie 10768785092763207838 [ 4187.663677] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4187.675550] LustreError: 126850:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4188.129653] Lustre: server umount lustre-MDT0001 complete [ 4202.352247] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4202.933158] LustreError: 130673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4202.950928] LustreError: 130673:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4203.030740] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4208.110160] LustreError: 130674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4208.125234] LustreError: 130674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4208.277269] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4212.194596] LustreError: 130673:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4217.318269] LustreError: 130674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4220.174610] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4220.891638] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4226.434317] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4235.369411] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4240.451852] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4247.758571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4248.317996] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4255.978967] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4263.030331] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4268.313981] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4274.076177] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 5 sec [ 4279.406786] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4279.660212] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4288.775870] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4289.031644] Lustre: lustre-OST0001: new disk, initializing [ 4289.036206] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4289.042550] Lustre: 133720:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4289.118724] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4295.567995] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4297.801473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4297.812604] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4297.913503] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4305.605512] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 4305.887704] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4310.243589] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 4310.442097] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4314.289519] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9da0582e7000.ost_server_uuid 50 [ 4316.306730] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9da0582e7000.ost_server_uuid in FULL state after 0 sec [ 4328.937047] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4328.938951] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4328.951171] LustreError: Skipped 1 previous similar message [ 4328.954384] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4328.993968] Lustre: Skipped 2 previous similar messages [ 4333.166353] Lustre: server umount lustre-OST0001 complete [ 4334.062905] LustreError: 132281:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4334.099168] LustreError: 132281:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4339.175431] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.184329] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4339.196695] Lustre: Skipped 1 previous similar message [ 4339.210162] Lustre: Skipped 2 previous similar messages [ 4344.054564] Lustre: server umount lustre-OST0000 complete [ 4348.898090] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4348.909578] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4355.262988] Lustre: server umount lustre-MDT0000 complete [ 4359.649828] LustreError: 130674:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4359.681204] LustreError: 130674:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4360.093277] LustreError: 133719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780339689 with bad export cookie 10768785092763208762 [ 4360.112281] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4360.112840] LustreError: 133719:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4360.846926] Lustre: server umount lustre-MDT0001 complete [ 4385.731244] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 4389.170768] Key type lgssc unregistered [ 4389.503816] LNet: 136558:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4389.509208] LNetError: 136558:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4390.573834] LNet: Removed LNI 192.168.202.107@tcp [ 4391.813728] Key type .llcrypt unregistered [ 4391.815809] Key type ._llcrypt unregistered [ 4435.782432] Key type ._llcrypt registered [ 4435.784777] Key type .llcrypt registered [ 4435.884285] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 4452.637936] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 4454.240993] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4454.344100] alg: No test for adler32 (adler32-zlib) [ 4455.607468] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 4455.866058] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 4457.562449] Key type lgssc registered [ 4458.807341] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4502.341442] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4502.360992] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4503.736908] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4503.784866] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4503.936832] Lustre: lustre-MDT0000: new disk, initializing [ 4504.100873] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4504.139393] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4510.105500] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4523.513260] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4523.646788] Lustre: 140680:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4523.739116] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4523.743471] Lustre: Skipped 1 previous similar message [ 4523.835534] Lustre: lustre-MDT0001: new disk, initializing [ 4523.948806] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4523.996974] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4524.012412] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4529.769604] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4539.737409] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4546.012307] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4555.961179] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4556.325368] Lustre: lustre-OST0000: new disk, initializing [ 4556.339417] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4556.354384] Lustre: 142427:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4556.491266] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4562.008344] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4562.019985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4562.162786] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4565.546509] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4575.587512] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4581.416501] Lustre: Failing over lustre-OST0000 [ 4582.881525] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4582.884367] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.900393] LustreError: Skipped 1 previous similar message [ 4582.923919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4583.669376] Lustre: server umount lustre-OST0000 complete [ 4605.002823] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4605.245691] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4605.262392] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 4610.542487] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 4614.166688] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4618.691944] Lustre: Failing over lustre-MDT0000 [ 4618.731320] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4618.742452] Lustre: Skipped 2 previous similar messages [ 4618.746296] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4618.772435] Lustre: Skipped 3 previous similar messages [ 4619.011397] Lustre: server umount lustre-MDT0000 complete [ 4625.896134] LustreError: 140689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4625.925152] LustreError: 140689:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 4627.940939] LustreError: 141845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4627.963992] LustreError: 141845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4631.007295] LustreError: 142640:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.075582] LustreError: 140693:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4633.123113] LustreError: 140693:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4637.212170] LustreError: 140688:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4637.234939] LustreError: 140688:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4639.009124] Lustre: 138904:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780339951/real 1780339951] req@00000000dc796123 x1866821569401088/t0(0) o400->MGC192.168.202.107@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1780339967 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 4639.040676] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4639.261722] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4647.402173] LustreError: 140689:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4647.431343] LustreError: 140689:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 4649.697855] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 4649.772221] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4649.822940] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 4652.526499] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 4655.106434] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4655.129691] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 4655.220457] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4655.574803] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4665.831373] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4665.847965] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4665.863118] LustreError: Skipped 1 previous similar message [ 4665.869052] Lustre: Skipped 1 previous similar message [ 4665.871951] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4670.435167] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4670.442652] Lustre: Skipped 2 previous similar messages [ 4671.742599] Lustre: server umount lustre-OST0000 complete [ 4680.673983] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4680.678605] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4680.704415] Lustre: Skipped 2 previous similar messages [ 4680.705428] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4680.727555] Lustre: Skipped 1 previous similar message [ 4682.842209] Lustre: server umount lustre-MDT0000 complete [ 4685.795206] LustreError: 141845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4685.819607] LustreError: 141845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 4687.237383] LustreError: 141462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340016 with bad export cookie 753687958336480021 [ 4687.242144] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4687.245556] LustreError: 141462:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4687.585952] Lustre: server umount lustre-MDT0001 complete [ 4700.539162] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 4703.920523] Key type lgssc unregistered [ 4704.293602] LNet: 145922:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4704.304897] LNetError: 145922:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4704.327160] LNet: Removed LNI 192.168.202.107@tcp [ 4705.120647] Key type .llcrypt unregistered [ 4705.122977] Key type ._llcrypt unregistered [ 4748.898370] Key type ._llcrypt registered [ 4748.901467] Key type .llcrypt registered [ 4749.037238] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 4750.823748] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4750.996928] alg: No test for adler32 (adler32-zlib) [ 4752.298828] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 4752.745125] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 4754.488178] Key type lgssc registered [ 4756.218917] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4765.784183] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4765.819646] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4767.563219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4772.672959] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4777.961606] LustreError: 147428:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4777.979666] LustreError: 147428:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4781.550631] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4783.074800] LustreError: 147429:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4786.224732] LustreError: 147457:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 4786.240227] LustreError: 147457:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 4786.253563] LustreError: 147457:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 19, retries 0, failed: rc = -5 [ 4786.711936] Lustre: server umount lustre-MDT0000 complete [ 4804.751506] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 4816.835516] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 4863.196070] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4863.575381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4863.617429] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4863.757590] Lustre: lustre-MDT0000: new disk, initializing [ 4863.856608] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4863.881902] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4869.243307] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4882.447930] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4882.649075] Lustre: 150617:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4882.753358] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4882.764175] Lustre: Skipped 1 previous similar message [ 4882.823616] Lustre: lustre-MDT0001: new disk, initializing [ 4882.907283] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4882.960641] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4882.967069] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4886.404872] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4897.136645] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4905.127129] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4917.707451] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4918.428567] Lustre: lustre-OST0000: new disk, initializing [ 4918.431930] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4918.436954] Lustre: 152363:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4918.509176] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4926.783972] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 4927.064416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4927.079692] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4927.308503] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4937.100652] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4942.184886] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 4944.569511] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 14:57:51 (1780340271) [ 5229.536685] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5229.537301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5229.556521] Lustre: Skipped 1 previous similar message [ 5229.574082] Lustre: Skipped 1 previous similar message [ 5234.151746] Lustre: server umount lustre-MDT0000 complete [ 5234.658298] LustreError: 150626:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5234.701543] LustreError: 150626:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5238.668578] LustreError: 151396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340567 with bad export cookie 13994331735331384836 [ 5238.672835] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5238.683770] LustreError: 151396:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5239.776752] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5239.780681] LustreError: 150625:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5239.792142] Lustre: Skipped 1 previous similar message [ 5239.795109] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5243.808816] LustreError: 150624:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5243.809370] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5245.066620] Lustre: server umount lustre-MDT0001 complete [ 5264.352210] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5264.627374] Lustre: server umount lustre-OST0000 complete [ 5277.095249] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 5290.986350] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 5341.232028] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5341.597839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5341.662698] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5341.737485] Lustre: lustre-MDT0000: new disk, initializing [ 5341.784702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5341.807643] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5347.471748] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 5361.253603] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5361.524741] Lustre: 156909:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5361.599661] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5361.602785] Lustre: Skipped 1 previous similar message [ 5361.725473] Lustre: lustre-MDT0001: new disk, initializing [ 5361.973266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5362.057973] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5362.069499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5368.200336] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 5378.086460] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5383.360453] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5392.562745] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5392.956609] Lustre: lustre-OST0000: new disk, initializing [ 5392.963377] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5392.977471] Lustre: 158655:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5393.070826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5398.617051] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5398.622569] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5398.826710] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5400.606187] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 5409.389411] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5414.575449] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5414.930400] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5420.854735] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5421.095972] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5424.611260] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5424.613088] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5424.622054] LustreError: Skipped 1 previous similar message [ 5424.642039] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5429.743179] LustreError: 158638:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5429.901909] Lustre: server umount lustre-OST0000 complete [ 5434.851555] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5434.868718] Lustre: Skipped 1 previous similar message [ 5434.874070] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5434.878172] Lustre: Skipped 1 previous similar message [ 5441.054045] Lustre: server umount lustre-MDT0000 complete [ 5446.898484] LustreError: 156902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340775 with bad export cookie 13994331735331543715 [ 5446.902492] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5446.916962] LustreError: 156902:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5447.873522] Lustre: server umount lustre-MDT0001 complete [ 5460.876167] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:06:28 (1780340788) [ 5479.822973] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 5493.686242] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 5542.721789] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5542.893573] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5542.911592] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5542.965623] Lustre: lustre-MDT0000: new disk, initializing [ 5543.043223] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5543.073861] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5547.865954] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 5561.739638] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5561.872366] Lustre: 163233:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5562.479327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5569.601268] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 5578.281824] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5583.697665] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5591.790445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5592.156695] Lustre: lustre-OST0000: new disk, initializing [ 5592.160519] Lustre: Skipped 1 previous similar message [ 5592.165893] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5592.168549] Lustre: Skipped 2 previous similar messages [ 5592.176688] Lustre: 164989:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5598.321062] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5598.333707] Lustre: Skipped 1 previous similar message [ 5598.336480] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5598.512951] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5600.772099] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 5610.194357] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5624.291573] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5624.302156] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5624.305623] LustreError: Skipped 1 previous similar message [ 5624.306670] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5624.306675] Lustre: Skipped 2 previous similar messages [ 5624.353345] Lustre: Skipped 2 previous similar messages [ 5625.736683] Lustre: server umount lustre-OST0000 complete [ 5634.017372] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5634.026119] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5637.092059] Lustre: server umount lustre-MDT0000 complete [ 5639.669288] LustreError: 163244:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5639.700704] LustreError: 163244:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5642.545455] LustreError: 165695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780340971 with bad export cookie 13994331735331544639 [ 5642.547663] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5642.560600] LustreError: 165695:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5643.071337] Lustre: server umount lustre-MDT0001 complete [ 5655.021856] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 15:09:42 (1780340982) [ 5676.150905] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 5687.754244] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 5739.730370] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5739.933792] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5739.953577] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5740.018757] Lustre: lustre-MDT0000: new disk, initializing [ 5740.079193] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5740.083368] Lustre: Skipped 2 previous similar messages [ 5740.100870] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5745.908184] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 5758.885181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5759.000788] Lustre: 169173:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5759.314838] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5765.213453] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 5775.645210] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5782.252868] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5791.584596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5791.955586] Lustre: 170927:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5800.856895] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 5801.103210] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5801.378095] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5810.230733] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5821.930374] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5821.935395] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5821.959187] LustreError: Skipped 1 previous similar message [ 5821.988116] Lustre: Skipped 1 previous similar message [ 5821.995289] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5822.024276] Lustre: Skipped 4 previous similar messages [ 5825.813774] Lustre: server umount lustre-OST0000 complete [ 5837.876429] Lustre: server umount lustre-MDT0000 complete [ 5842.418826] LustreError: 169179:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5842.448270] LustreError: 169179:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5842.658745] LustreError: 169164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341171 with bad export cookie 13994331735331545766 [ 5842.667140] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5842.672915] LustreError: 169164:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5843.188852] Lustre: server umount lustre-MDT0001 complete [ 5855.457613] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 5859.034652] Key type lgssc unregistered [ 5859.345068] LNet: 172657:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5859.355469] LNetError: 172657:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5859.381912] LNet: Removed LNI 192.168.202.107@tcp [ 5860.227577] Key type .llcrypt unregistered [ 5860.229928] Key type ._llcrypt unregistered [ 5889.380379] Key type ._llcrypt registered [ 5889.390803] Key type .llcrypt registered [ 5889.593315] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 5891.030519] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5891.094596] alg: No test for adler32 (adler32-zlib) [ 5892.251216] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 5892.476459] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 5894.256366] Key type lgssc registered [ 5895.464131] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5905.051913] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5905.072832] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5906.704652] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5911.192639] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 5916.129403] LustreError: 174060:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5916.149877] LustreError: 174060:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5920.505872] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5920.901962] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5925.629488] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 5931.859890] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5936.421783] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5943.754754] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5944.115227] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5950.171944] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 5951.287433] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 5956.371860] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5976.036568] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5976.044807] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5976.060765] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5977.059133] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5977.074276] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5977.083219] Lustre: Skipped 1 previous similar message [ 5979.799917] Lustre: server umount lustre-OST0000 complete [ 5987.298429] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5987.303476] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5987.315393] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5987.341654] Lustre: Skipped 1 previous similar message [ 5990.580955] Lustre: server umount lustre-MDT0000 complete [ 5992.428536] LustreError: 174786:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5992.460657] LustreError: 174786:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5994.010104] LustreError: 174041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341322 with bad export cookie 13385501361428117547 [ 5994.020904] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5994.337858] Lustre: server umount lustre-MDT0001 complete [ 6006.632339] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 6009.283485] Key type lgssc unregistered [ 6009.658215] LNet: 177259:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6009.667586] LNetError: 177259:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6009.689826] LNet: Removed LNI 192.168.202.107@tcp [ 6011.104661] Key type .llcrypt unregistered [ 6011.108442] Key type ._llcrypt unregistered [ 6034.837562] Key type ._llcrypt registered [ 6034.839631] Key type .llcrypt registered [ 6034.922295] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 6036.123802] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6036.165845] alg: No test for adler32 (adler32-zlib) [ 6037.284534] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 6037.567920] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 6039.312220] Key type lgssc registered [ 6040.584220] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6049.181931] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6049.203344] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6050.885068] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6055.639879] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6061.031855] LustreError: 178660:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6061.046607] LustreError: 178660:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6064.959987] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6065.361173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6070.505207] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6076.466343] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6079.692870] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6086.649574] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6087.088080] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6094.096828] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6096.372440] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6100.059878] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6108.162791] Lustre: lustre-OST0000: Not available for connect from 192.168.202.7@tcp (stopping) [ 6110.693617] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6110.699390] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6110.722505] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6111.724669] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6111.727276] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6111.756811] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6112.917784] Lustre: server umount lustre-OST0000 complete [ 6118.693247] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff9da0454d9800.ost_server_uuid 50 [ 6120.561564] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 6136.289989] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6136.290919] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6136.298650] Lustre: Skipped 1 previous similar message [ 6136.314984] Lustre: Skipped 2 previous similar messages [ 6140.671203] Lustre: server umount lustre-MDT0000 complete [ 6141.409779] LustreError: 179623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6141.434565] LustreError: 179623:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6143.911888] LustreError: 178641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341472 with bad export cookie 11955863972619145843 [ 6143.917658] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6143.923270] LustreError: 178641:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6144.303254] Lustre: server umount lustre-MDT0001 complete [ 6151.797197] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 15:17:59 (1780341479) [ 6158.006627] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6158.452042] LustreError: 182137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6158.537491] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6162.219552] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6163.944424] LustreError: 182138:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6168.034809] LustreError: 182137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6170.390793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6170.756534] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6174.681576] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6180.068866] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6183.410945] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6189.999304] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6190.286102] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6196.223501] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6201.477531] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6203.648414] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6207.668237] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6207.838680] Lustre: lustre-OST0001: new disk, initializing [ 6207.841384] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6207.845446] Lustre: 184639:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6207.915918] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 6213.331640] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6217.804357] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6217.818946] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6217.883908] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6220.594140] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6224.354173] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6224.365269] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6224.385995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6226.406021] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6229.887942] Lustre: server umount lustre-OST0000 complete [ 6231.524246] LustreError: 183737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6236.946652] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6237.193992] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6242.452947] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6242.694612] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6248.940668] LustreError: 183737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6248.961153] LustreError: 183737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 10 previous similar messages [ 6259.175086] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6259.184463] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6259.191188] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6259.192549] Lustre: Skipped 2 previous similar messages [ 6259.203050] Lustre: Skipped 5 previous similar messages [ 6262.098928] Lustre: server umount lustre-OST0001 complete [ 6267.874798] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6267.884505] LustreError: Skipped 1 previous similar message [ 6267.890103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6272.010553] Lustre: server umount lustre-MDT0000 complete [ 6275.642542] LustreError: 182120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341604 with bad export cookie 11955863972619146907 [ 6275.653413] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6275.654044] LustreError: 182120:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6275.880407] Lustre: server umount lustre-MDT0001 complete [ 6296.586393] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 15:20:24 (1780341624) [ 6303.120237] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6303.398953] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6303.432995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6303.590035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6307.699875] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6315.942947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6316.019241] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6316.043685] Lustre: 187931:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6320.448743] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6326.818809] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6330.197761] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6336.549131] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6336.744596] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6343.382543] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6346.859587] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:129) [ 6348.728331] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6356.103794] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6356.223144] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6363.664419] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6364.896457] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:3 to 0x2c0000401:33) [ 6369.979821] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6372.834619] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6372.848442] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6372.871829] Lustre: Skipped 1 previous similar message [ 6372.875857] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6372.881162] Lustre: Skipped 3 previous similar messages [ 6378.851575] Lustre: server umount lustre-OST0000 complete [ 6380.521203] LustreError: 189557:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6380.544616] LustreError: 189557:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 6400.108657] Lustre: server umount lustre-OST0001 complete [ 6406.114827] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6406.116719] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6406.125098] Lustre: Skipped 6 previous similar messages [ 6406.148715] Lustre: Skipped 4 previous similar messages [ 6409.859291] Lustre: server umount lustre-MDT0000 complete [ 6413.357154] LustreError: 187922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341742 with bad export cookie 11955863972619148321 [ 6413.366378] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6413.369728] LustreError: 187922:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 6413.721222] Lustre: server umount lustre-MDT0001 complete [ 6433.916478] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 15:22:42 (1780341762) [ 6447.196896] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 6455.112273] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 6485.486130] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6485.719377] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6485.759473] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6485.850725] Lustre: lustre-MDT0000: new disk, initializing [ 6485.958115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6485.968860] Lustre: Skipped 3 previous similar messages [ 6485.994390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6489.203049] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6497.247882] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6497.318496] Lustre: 195537:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6497.353244] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6497.354643] Lustre: Skipped 1 previous similar message [ 6497.417480] Lustre: lustre-MDT0001: new disk, initializing [ 6497.539622] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6497.547542] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6500.045559] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6505.602195] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6508.187543] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6513.602462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6513.825775] Lustre: lustre-OST0000: new disk, initializing [ 6513.829652] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6513.837685] Lustre: 197280:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6518.894133] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6522.486068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6522.491464] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6522.559495] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6525.809527] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6529.987699] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6530.206262] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6533.328895] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6533.504869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6538.209479] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6538.218172] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6538.220595] LustreError: Skipped 3 previous similar messages [ 6538.229585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6538.235773] Lustre: Skipped 2 previous similar messages [ 6541.102291] Lustre: server umount lustre-OST0000 complete [ 6549.816531] Lustre: server umount lustre-MDT0000 complete [ 6552.927533] LustreError: 196316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780341881 with bad export cookie 11955863972619150099 [ 6552.927897] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6552.947521] LustreError: 196316:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 6553.170428] Lustre: server umount lustre-MDT0001 complete [ 6560.231203] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6560.506484] LustreError: 199348:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6560.517449] LustreError: 199348:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 6563.954189] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6571.296216] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6575.956360] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6580.596563] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6584.045635] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6589.608943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6594.530526] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6598.545946] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6601.522896] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6606.016859] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 4 sec [ 6609.943624] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6610.102806] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6615.528931] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6615.537939] LustreError: Skipped 1 previous similar message [ 6617.790397] Lustre: server umount lustre-OST0000 complete [ 6622.766915] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6626.213375] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6653.035210] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6653.296415] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6653.301080] Lustre: Skipped 5 previous similar messages [ 6655.356945] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6655.368909] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6655.374014] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6655.395255] Lustre: Skipped 1 previous similar message [ 6657.512434] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6661.796473] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6674.421263] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 10 sec [ 6681.058291] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.063413] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6681.068571] Lustre: Skipped 6 previous similar messages [ 6681.076474] Lustre: Skipped 6 previous similar messages [ 6692.320452] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6692.484981] Lustre: server umount lustre-OST0000 complete [ 6701.799476] Lustre: server umount lustre-MDT0000 complete [ 6704.687448] LustreError: 202114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342033 with bad export cookie 11955863972619151030 [ 6704.694238] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6704.695163] LustreError: 202114:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6705.072885] Lustre: server umount lustre-MDT0001 complete [ 6711.675392] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 15:27:19 (1780342039) [ 6717.016337] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6717.373913] LustreError: 204846:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6717.386835] LustreError: 204846:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6720.488242] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6727.272940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6730.793828] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6734.876575] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6737.941585] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6743.268170] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6747.563919] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6751.142416] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6754.284834] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6754.438166] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6757.885683] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6758.059743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6762.648801] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6762.753261] Lustre: lustre-OST0001: new disk, initializing [ 6762.754820] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6762.757376] Lustre: 207841:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6767.016502] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6768.205334] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6768.213071] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6768.280937] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6772.954446] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6776.194431] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6776.368895] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6779.258345] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6779.381156] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6783.970980] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6783.979064] LustreError: Skipped 2 previous similar messages [ 6786.673390] Lustre: server umount lustre-OST0001 complete [ 6791.447374] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6791.613980] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 6794.159210] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6794.343208] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 6820.952555] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6825.562225] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6826.477168] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6826.485908] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 6826.486614] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6826.501988] Lustre: Skipped 1 previous similar message [ 6829.073433] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6830.141420] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6837.534961] Lustre: server umount lustre-OST0001 complete [ 6858.791628] Lustre: server umount lustre-OST0000 complete [ 6867.990671] Lustre: server umount lustre-MDT0000 complete [ 6870.686538] LustreError: 204826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342199 with bad export cookie 11955863972619151996 [ 6870.688691] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6870.693259] LustreError: 204826:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6876.993437] Lustre: server umount lustre-MDT0001 complete [ 6892.473121] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 15:30:20 (1780342220) [ 6896.999428] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6897.199098] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6897.213737] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6900.303599] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6906.252427] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6906.310489] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6906.328509] Lustre: 212785:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6909.574975] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6913.907704] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6916.294073] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6920.812787] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6920.949485] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6921.078225] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6921.084028] Lustre: Skipped 7 previous similar messages [ 6924.855911] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6928.360573] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6934.800910] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6934.883535] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 6938.459482] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 6941.773533] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6943.067234] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6945.788439] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 6945.949992] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6948.679207] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 6948.852790] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6951.256995] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff9da0481e0000.ost_server_uuid 50 [ 6952.406819] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff9da0481e0000.ost_server_uuid in FULL state after 0 sec [ 6953.065300] Lustre: Permanently deactivating lustre-OST0001 [ 6953.085293] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 6954.928323] Lustre: Permanently reactivating lustre-OST0001 [ 6959.585102] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6959.591781] Lustre: Skipped 10 previous similar messages [ 6959.594824] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6959.597243] Lustre: Skipped 22 previous similar messages [ 6963.487514] Lustre: server umount lustre-OST0001 complete [ 6972.441477] Lustre: server umount lustre-OST0000 complete [ 6989.281033] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6989.411166] Lustre: server umount lustre-MDT0000 complete [ 6990.817593] LustreError: 213526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6990.823631] LustreError: 213526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 34 previous similar messages [ 6991.494272] LustreError: 212775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342320 with bad export cookie 11955863972619153284 [ 6991.499307] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6991.500979] LustreError: 212775:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6991.704271] Lustre: server umount lustre-MDT0001 complete [ 7005.750281] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 15:32:14 (1780342334) [ 7010.084530] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7010.250257] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7010.265688] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7010.268841] Lustre: Skipped 5 previous similar messages [ 7012.603098] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7017.923606] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7018.026364] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7018.061888] Lustre: 218890:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7020.446308] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7023.655413] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7025.745851] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7029.459632] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7029.575544] Lustre: Permanently deactivating lustre-OST0000 [ 7029.581841] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 7032.375695] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7035.069565] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7038.756548] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7038.891413] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 7038.968387] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 7041.987258] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7045.029639] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7045.942508] Lustre: Permanently reactivating lustre-OST0000 [ 7045.951621] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 7045.954546] Lustre: Skipped 2 previous similar messages [ 7055.877405] Lustre: lustre-OST0000: Client lustre-MDT0001-mdtlov_UUID (at 0@lo) reconnecting [ 7055.880966] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7055.889983] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7055.915079] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 7055.923313] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7057.897680] LustreError: lustre-OST0001-osc-MDT0001: operation ost_destroy to node 0@lo failed: rc = -107 [ 7057.904051] LustreError: Skipped 9 previous similar messages [ 7062.092535] Lustre: server umount lustre-OST0001 complete [ 7070.504235] Lustre: server umount lustre-OST0000 complete [ 7078.716196] Lustre: server umount lustre-MDT0000 complete [ 7080.920207] LustreError: 218882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342409 with bad export cookie 11955863972619154936 [ 7080.922807] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7080.925012] LustreError: 218882:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7081.118859] Lustre: server umount lustre-MDT0001 complete [ 7087.566618] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 7089.169933] Key type lgssc unregistered [ 7089.371567] LNet: 223272:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7089.375492] LNetError: 223272:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7089.384694] LNet: Removed LNI 192.168.202.107@tcp [ 7090.071856] Key type .llcrypt unregistered [ 7090.075470] Key type ._llcrypt unregistered [ 7105.625468] Key type ._llcrypt registered [ 7105.629847] Key type .llcrypt registered [ 7105.690919] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7106.352797] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7106.362910] alg: No test for adler32 (adler32-zlib) [ 7107.296227] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 7107.453107] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 7109.064204] Key type lgssc registered [ 7109.816249] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7115.918894] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7115.929965] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7117.237879] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7119.829855] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7124.765850] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7124.840690] Lustre: Found index 1 for lustre-MDT0001, updating log [ 7124.856716] Lustre: 224690:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7124.863896] Lustre: Permanently deactivating lustre-MDT0001 [ 7124.873839] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7124.917303] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7124.927439] LustreError: 224731:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 7124.936581] LustreError: 224731:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 7124.948895] LustreError: 224731:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 1, failed: rc = -108 [ 7125.072529] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7127.346721] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7130.930221] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7133.257303] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7137.419349] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7137.606097] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7140.458373] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7142.758827] LustreError: 226315:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7142.769117] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 7142.770708] LustreError: 226315:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7143.294973] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7147.183873] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7147.309784] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7149.355927] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 7149.357705] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 7150.572237] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7154.071377] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7156.212521] Lustre: Permanently reactivating lustre-MDT0001 [ 7156.223151] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 7156.227183] Lustre: Skipped 2 previous similar messages [ 7161.850149] Lustre: lustre-MDT0001: Client cfbb131f-b905-4f24-b544-d2fc5ff2b703 (at 192.168.202.7@tcp) reconnecting [ 7164.773353] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7165.838533] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 7168.363770] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 7168.477748] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 7170.537337] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 7170.631648] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 7171.932492] Lustre: Permanently deactivating lustre-MDT0001 [ 7171.940916] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 7171.944494] Lustre: Skipped 2 previous similar messages [ 7179.799355] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 7186.401057] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7186.404048] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7186.415112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7189.474379] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7189.475195] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7189.487693] Lustre: Skipped 2 previous similar messages [ 7189.499044] Lustre: Skipped 2 previous similar messages [ 7190.937902] LustreError: 228777:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x736d61726170:0x3:0x0].0x0 (00000000bf57dffa) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 7193.031414] Lustre: server umount lustre-MDT0000 complete [ 7194.596391] LustreError: 224700:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7194.608282] LustreError: 224700:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7195.211343] LustreError: 224681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342524 with bad export cookie 9280260326271143143 [ 7195.213918] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7195.225978] LustreError: 224681:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7195.449549] Lustre: server umount lustre-MDT0001 complete [ 7204.079742] Lustre: server umount lustre-OST0000 complete [ 7210.759391] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 15:35:39 (1780342539) [ 7217.478971] Lustre: server umount lustre-OST0001 complete [ 7221.501861] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 7226.247941] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7244.710493] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7244.901936] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7244.907536] Lustre: Skipped 2 previous similar messages [ 7244.926984] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7244.982388] Lustre: lustre-MDT0000: new disk, initializing [ 7245.037777] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7245.046882] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7247.365756] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7253.578522] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7253.648211] Lustre: 232355:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7253.692069] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7253.696779] Lustre: Skipped 1 previous similar message [ 7253.736519] Lustre: lustre-MDT0001: new disk, initializing [ 7253.780990] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7253.801045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7253.805649] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7256.100297] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7260.467207] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7262.325458] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7265.535919] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7265.647405] Lustre: lustre-OST0000: new disk, initializing [ 7265.649541] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7265.652655] Lustre: 234098:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7266.972975] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7266.978171] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7267.029990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7268.501906] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7272.771521] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7274.301781] LustreError: 232364:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7280.007778] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7280.096378] Lustre: lustre-OST0001: new disk, initializing [ 7280.099338] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 7280.103318] Lustre: 235100:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7280.147031] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 7280.152600] Lustre: Skipped 1 previous similar message [ 7283.201281] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7284.328109] LustreError: 232364:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7284.338358] LustreError: 234827:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 7286.286346] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 7286.291136] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 7286.319747] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 7287.465290] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 7288.280279] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 7294.352135] LustreError: 234827:0:(mdt_handler.c:3264:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 7296.481579] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7296.489632] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7296.499675] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 7296.505949] Lustre: Skipped 2 previous similar messages [ 7301.448821] Lustre: server umount lustre-OST0001 complete [ 7301.604987] LustreError: 234081:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7301.610024] LustreError: 234081:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7306.723890] LustreError: 234082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7306.729788] LustreError: 234082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7311.847797] LustreError: 234082:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7311.853172] LustreError: 234082:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7316.963633] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7316.965966] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7316.971017] Lustre: Skipped 2 previous similar messages [ 7316.973412] Lustre: Skipped 1 previous similar message [ 7320.667950] Lustre: server umount lustre-OST0000 complete [ 7325.664895] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7325.667805] LustreError: Skipped 1 previous similar message [ 7325.669315] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7325.676188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7325.678858] Lustre: Skipped 1 previous similar message [ 7328.494812] Lustre: server umount lustre-MDT0000 complete [ 7330.216221] LustreError: 232349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342659 with bad export cookie 9280260326271145747 [ 7330.219362] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7330.221818] LustreError: 232349:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 7336.425496] Lustre: server umount lustre-MDT0001 complete [ 7340.920705] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 7342.209356] Key type lgssc unregistered [ 7342.356685] LNet: 236999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7342.360367] LNetError: 236999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7342.370373] LNet: Removed LNI 192.168.202.107@tcp [ 7342.718615] Key type .llcrypt unregistered [ 7342.720298] Key type ._llcrypt unregistered [ 7362.066162] Key type ._llcrypt registered [ 7362.067243] Key type .llcrypt registered [ 7362.115930] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7362.761078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7362.778073] alg: No test for adler32 (adler32-zlib) [ 7363.682582] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 7363.781917] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 7365.384159] Key type lgssc registered [ 7365.818538] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7368.805732] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7368.813913] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7369.928722] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7369.941233] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7370.019970] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7371.536535] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7374.928513] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7374.988519] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7375.002890] Lustre: 239167:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7375.119705] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7376.825928] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7379.141316] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7380.646491] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7383.776141] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7383.858212] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7383.932770] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7386.234326] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 7386.612880] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7389.481743] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7432.160944] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7432.164357] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.169953] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7432.672923] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7432.675891] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7432.682122] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7437.794971] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7437.799022] Lustre: Skipped 1 previous similar message [ 7438.196113] Lustre: server umount lustre-OST0000 complete [ 7440.471956] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 7446.130321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7446.305123] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7448.768476] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7451.099195] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7462.693767] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7462.704618] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 7462.706612] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 7488.483097] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7488.484132] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7488.485487] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7488.499069] LustreError: Skipped 1 previous similar message [ 7488.507682] Lustre: Skipped 1 previous similar message [ 7491.972726] Lustre: server umount lustre-OST0000 complete [ 7498.208746] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7498.211593] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7498.217757] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7498.221030] Lustre: Skipped 1 previous similar message [ 7500.199464] Lustre: server umount lustre-MDT0000 complete [ 7501.840606] LustreError: 239157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342830 with bad export cookie 12094430082888811161 [ 7501.843856] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7501.845070] LustreError: 239157:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7502.034650] Lustre: server umount lustre-MDT0001 complete [ 7510.510225] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 7512.899226] Key type lgssc unregistered [ 7513.115806] LNet: 244115:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7513.122295] LNetError: 244115:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7513.132882] LNet: Removed LNI 192.168.202.107@tcp [ 7513.734484] Key type .llcrypt unregistered [ 7513.736628] Key type ._llcrypt unregistered [ 7540.788347] Key type ._llcrypt registered [ 7540.795663] Key type .llcrypt registered [ 7540.942902] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7542.308292] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7542.477553] alg: No test for adler32 (adler32-zlib) [ 7544.097795] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 7544.365348] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 7546.098936] Key type lgssc registered [ 7547.481512] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7555.675562] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7555.696587] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7557.357942] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7561.275609] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7567.843562] LustreError: 245516:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7567.865905] LustreError: 245516:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7572.965712] LustreError: 245515:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7573.072591] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7573.624621] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7578.921388] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7586.686800] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7590.189086] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7595.857525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7596.181575] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7601.223251] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7602.350589] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 7605.179222] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7623.137622] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7623.142062] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7623.153488] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7623.156327] Lustre: Skipped 1 previous similar message [ 7623.162763] Lustre: Skipped 1 previous similar message [ 7627.496916] Lustre: server umount lustre-OST0000 complete [ 7633.377898] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7633.395433] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7633.404533] Lustre: Skipped 1 previous similar message [ 7636.752415] Lustre: server umount lustre-MDT0000 complete [ 7638.502610] LustreError: 245512:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7638.533505] LustreError: 245512:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7639.667948] LustreError: 245497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780342968 with bad export cookie 10714196941257879775 [ 7639.673722] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7639.677505] LustreError: 245497:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7640.081630] Lustre: server umount lustre-MDT0001 complete [ 7649.107422] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 7651.333609] Key type lgssc unregistered [ 7651.591628] LNet: 249464:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7651.601778] LNetError: 249464:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7652.647114] LNet: Removed LNI 192.168.202.107@tcp [ 7653.344266] Key type .llcrypt unregistered [ 7653.346345] Key type ._llcrypt unregistered [ 7668.303352] Key type ._llcrypt registered [ 7668.310574] Key type .llcrypt registered [ 7668.429852] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7669.375730] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7669.421738] alg: No test for adler32 (adler32-zlib) [ 7670.467326] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 7670.671725] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 7672.360225] Key type lgssc registered [ 7673.296508] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7679.390178] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7679.407221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7680.879702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7685.057868] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7690.210791] LustreError: 250666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7690.225157] LustreError: 250666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 7691.750547] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7691.994670] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7695.128647] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7699.329759] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7702.561380] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7708.350742] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7708.633105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7713.476789] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7717.366503] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7726.063598] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 7736.290074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7736.292178] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7736.301021] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7736.309422] Lustre: Skipped 1 previous similar message [ 7741.414790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7741.419076] Lustre: Skipped 1 previous similar message [ 7742.080769] Lustre: server umount lustre-OST0000 complete [ 7746.530199] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7746.539435] Lustre: Skipped 1 previous similar message [ 7746.542154] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7746.549477] Lustre: Skipped 1 previous similar message [ 7751.650201] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7751.655805] Lustre: Skipped 1 previous similar message [ 7756.771899] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7756.777322] Lustre: Skipped 1 previous similar message [ 7759.328723] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7759.439495] Lustre: server umount lustre-MDT0000 complete [ 7761.891317] LustreError: 250666:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7761.907929] LustreError: 250666:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7762.381382] LustreError: 250645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343091 with bad export cookie 4875704012014548701 [ 7762.385876] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7762.392031] LustreError: 250645:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7762.705786] Lustre: server umount lustre-MDT0001 complete [ 7770.457423] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 7772.513969] Key type lgssc unregistered [ 7772.728514] LNet: 254008:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7772.737056] LNetError: 254008:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7772.771879] LNet: Removed LNI 192.168.202.107@tcp [ 7773.383475] Key type .llcrypt unregistered [ 7773.385216] Key type ._llcrypt unregistered [ 7790.372994] Key type ._llcrypt registered [ 7790.374533] Key type .llcrypt registered [ 7790.448664] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7791.075674] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7791.099513] alg: No test for adler32 (adler32-zlib) [ 7792.125192] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 7792.311649] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 7793.944388] Key type lgssc registered [ 7794.800958] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7800.689880] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7800.711209] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7802.030061] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7805.266448] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7811.140241] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7811.391555] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7814.911802] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7819.475617] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7822.436462] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7828.035338] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7828.394743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7833.615837] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7838.108592] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7847.851766] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 7872.483786] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7872.493108] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7872.504631] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7873.506970] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7873.513314] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7873.514545] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7875.697702] Lustre: server umount lustre-OST0000 complete [ 7883.237694] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7883.253189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7883.264918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7883.280464] Lustre: Skipped 1 previous similar message [ 7886.156342] Lustre: server umount lustre-MDT0000 complete [ 7888.866422] LustreError: 255380:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7888.884432] LustreError: 255380:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 7890.089493] LustreError: 255366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343219 with bad export cookie 17709103122764448476 [ 7890.095994] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7890.100726] LustreError: 255366:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7890.649948] Lustre: server umount lustre-MDT0001 complete [ 7900.357462] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 7902.793429] Key type lgssc unregistered [ 7903.089710] LNet: 259356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7903.103383] LNetError: 259356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7903.117526] LNet: Removed LNI 192.168.202.107@tcp [ 7903.794277] Key type .llcrypt unregistered [ 7903.800304] Key type ._llcrypt unregistered [ 7916.935785] Key type ._llcrypt registered [ 7916.937878] Key type .llcrypt registered [ 7917.043887] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 7917.766447] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7917.887055] alg: No test for adler32 (adler32-zlib) [ 7918.844446] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 7919.007987] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 7920.672362] Key type lgssc registered [ 7921.649773] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7927.589782] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7927.607750] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7928.964074] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7931.982956] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7937.629596] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7937.893326] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7940.430959] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7943.914213] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7946.608941] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7951.240525] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7951.461376] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7953.542715] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7955.933834] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 7959.638360] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7973.352965] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7973.353721] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7973.373294] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7973.377669] Lustre: Skipped 1 previous similar message [ 7978.160929] Lustre: server umount lustre-OST0000 complete [ 7983.586240] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7983.597894] Lustre: Skipped 1 previous similar message [ 7983.600752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7988.706460] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7988.710821] Lustre: Skipped 2 previous similar messages [ 7993.826685] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7993.834722] Lustre: Skipped 1 previous similar message [ 7995.361454] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7995.493376] Lustre: server umount lustre-MDT0000 complete [ 7998.134340] LustreError: 260519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343327 with bad export cookie 706508188019196317 [ 7998.140610] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7998.143840] LustreError: 260519:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7998.463712] Lustre: server umount lustre-MDT0001 complete [ 8007.463646] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 8009.601735] Key type lgssc unregistered [ 8009.848941] LNet: 263903:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8009.854083] LNetError: 263903:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8009.866161] LNet: Removed LNI 192.168.202.107@tcp [ 8010.478352] Key type .llcrypt unregistered [ 8010.480304] Key type ._llcrypt unregistered [ 8019.141990] Key type ._llcrypt registered [ 8019.144364] Key type .llcrypt registered [ 8019.210944] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 8032.110753] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 8044.126314] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8045.232829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8045.261985] alg: No test for adler32 (adler32-zlib) [ 8046.283447] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 8046.444953] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 8048.080229] Key type lgssc registered [ 8048.953905] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8073.413887] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8073.444066] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8074.610628] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8074.633841] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8074.707325] Lustre: lustre-MDT0000: new disk, initializing [ 8074.778446] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8074.793167] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8078.042690] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8086.154670] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8086.224451] Lustre: 267379:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8086.265321] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8086.268902] Lustre: Skipped 1 previous similar message [ 8086.318897] Lustre: lustre-MDT0001: new disk, initializing [ 8086.381970] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8086.431568] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8086.447478] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8089.091181] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8094.286358] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8096.789618] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8101.481385] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8101.791311] Lustre: lustre-OST0000: new disk, initializing [ 8101.794119] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8101.798642] Lustre: 269122:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8101.876428] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8101.985560] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8101.994052] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8102.068507] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8106.016721] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8111.407708] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8114.528444] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8114.685868] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8117.454386] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8117.611844] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8122.859696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8122.860380] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8122.866594] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8122.874792] Lustre: Skipped 1 previous similar message [ 8122.890551] Lustre: Skipped 1 previous similar message [ 8125.116441] Lustre: server umount lustre-OST0000 complete [ 8127.465595] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8127.474320] LustreError: Skipped 1 previous similar message [ 8127.477125] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8127.486499] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8133.093342] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8133.099767] Lustre: Skipped 2 previous similar messages [ 8133.456595] Lustre: server umount lustre-MDT0000 complete [ 8136.363903] LustreError: 267371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343465 with bad export cookie 13942358383494568595 [ 8136.369219] LustreError: 267371:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8136.370808] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8136.706538] Lustre: server umount lustre-MDT0001 complete [ 8141.922972] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 15:51:10 (1780343470) [ 8145.856148] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8146.244301] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8148.967671] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8151.522283] LustreError: 271387:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8151.534427] LustreError: 271387:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8154.241539] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8154.426984] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8157.027762] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8160.376324] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8162.906567] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8167.492201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8167.714635] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8172.183461] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8175.823942] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8187.361918] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8187.369232] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8187.382889] Lustre: Skipped 1 previous similar message [ 8187.397803] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8190.439780] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8190.461900] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8192.780596] Lustre: server umount lustre-OST0000 complete [ 8195.506544] Lustre: server umount lustre-MDT0000 complete [ 8195.554027] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8195.562107] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8195.570679] LustreError: 272097:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8195.602753] Lustre: Skipped 1 previous similar message [ 8198.232814] LustreError: 271367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343527 with bad export cookie 13942358383494569519 [ 8198.241064] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8198.241845] LustreError: 271367:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8198.506735] Lustre: server umount lustre-MDT0001 complete [ 8204.925959] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 8206.512697] Key type lgssc unregistered [ 8206.681701] LNet: 274563:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8206.688753] LNetError: 274563:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8206.702824] LNet: Removed LNI 192.168.202.107@tcp [ 8207.222244] Key type .llcrypt unregistered [ 8207.225107] Key type ._llcrypt unregistered [ 8226.787983] Key type ._llcrypt registered [ 8226.790283] Key type .llcrypt registered [ 8226.865585] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 8237.513195] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8238.169503] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8238.177572] alg: No test for adler32 (adler32-zlib) [ 8239.145269] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 8239.278859] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 8240.880221] Key type lgssc registered [ 8241.695968] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8273.943648] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8273.960756] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8275.128436] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8275.154168] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8275.203963] Lustre: lustre-MDT0000: new disk, initializing [ 8275.257851] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8275.275149] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8277.602410] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8284.588136] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8284.668108] Lustre: 278330:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8284.698524] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8284.700874] Lustre: Skipped 1 previous similar message [ 8284.747122] Lustre: lustre-MDT0001: new disk, initializing [ 8284.819189] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8284.843066] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8284.847499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8287.849194] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8293.403853] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8296.432657] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8301.332616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8301.492935] Lustre: lustre-OST2710: new disk, initializing [ 8301.495897] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 8301.502058] Lustre: 280070:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8301.549720] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 8306.179131] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8307.770561] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 8307.793640] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 8307.885812] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 8312.400516] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 8317.788877] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8317.963866] Lustre: lustre-OST03e8: new disk, initializing [ 8317.966895] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 8317.974984] Lustre: 281024:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8318.050401] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 8322.580761] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8325.652993] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 8325.663867] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 8325.733393] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 8327.605659] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 8333.927610] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 8334.080640] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8337.304657] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 8337.456424] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8341.011689] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 8341.219771] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8344.361050] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 8344.515592] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8351.207674] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8351.213500] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8351.223939] Lustre: Skipped 2 previous similar messages [ 8351.227537] Lustre: Skipped 1 previous similar message [ 8353.884078] Lustre: server umount lustre-MDT0000 complete [ 8356.404095] LustreError: 278322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343685 with bad export cookie 11377957324743896552 [ 8356.407307] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8356.412729] LustreError: 278322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8356.631329] Lustre: server umount lustre-MDT0001 complete [ 8369.739086] Lustre: server umount lustre-OST2710 complete [ 8382.394200] Lustre: server umount lustre-OST03e8 complete [ 8394.391390] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 8401.700116] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8428.256419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8428.444184] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8428.464705] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8428.528235] Lustre: lustre-MDT0000: new disk, initializing [ 8428.591408] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8428.600465] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8431.742363] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8439.383140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8439.495609] Lustre: 286053:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8439.571340] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8439.577778] Lustre: Skipped 1 previous similar message [ 8439.652476] Lustre: lustre-MDT0001: new disk, initializing [ 8439.727453] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8439.754794] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8439.765261] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8442.775399] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8448.895761] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8452.047865] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8457.282814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8457.470936] Lustre: lustre-OST0000: new disk, initializing [ 8457.474829] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8457.480953] Lustre: 287797:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8457.539077] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8462.019656] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8465.965462] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8465.973094] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8466.074936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8468.046167] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8471.303494] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8471.451095] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8474.216376] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8474.355433] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8476.133630] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8476.140229] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8476.152190] Lustre: Skipped 1 previous similar message [ 8476.156101] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8476.159100] Lustre: Skipped 2 previous similar messages [ 8476.642355] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8481.770048] LustreError: 287781:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8481.780432] LustreError: 287781:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8481.914690] Lustre: server umount lustre-OST0000 complete [ 8485.857267] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8485.869767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8485.877120] Lustre: Skipped 1 previous similar message [ 8485.880202] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8485.882719] Lustre: Skipped 1 previous similar message [ 8490.780820] Lustre: server umount lustre-MDT0000 complete [ 8492.003437] LustreError: 286849:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8492.011479] LustreError: 286849:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8493.566738] LustreError: 286831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343822 with bad export cookie 11377957324743898281 [ 8493.567029] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8493.575069] LustreError: 286831:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8493.797910] Lustre: server umount lustre-MDT0001 complete [ 8499.709645] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 15:57:08 (1780343828) [ 8500.681107] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 8501.848071] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 15:57:10 (1780343830) [ 8503.583416] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8521.121937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8521.276592] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8521.296740] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8521.408104] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8523.868271] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8529.052314] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8529.113292] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8529.133515] Lustre: 291483:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8532.015135] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8550.827188] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8550.951933] Lustre: Denying initial registration attempt from nid 192.168.202.107@tcp, specified as failover [ 8550.956332] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 8550.961350] LustreError: 292907:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -99 [ 8550.965619] LustreError: 292907:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8550.968567] LustreError: 292907:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8551.017241] Lustre: server umount lustre-OST0000 complete [ 8551.019840] LustreError: 292907:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 8556.514162] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8556.518110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8556.521268] Lustre: Skipped 2 previous similar messages [ 8556.528099] Lustre: Skipped 3 previous similar messages [ 8561.636884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8561.642240] Lustre: Skipped 1 previous similar message [ 8562.303643] Lustre: server umount lustre-MDT0000 complete [ 8564.670482] LustreError: 291476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780343893 with bad export cookie 11377957324743899205 [ 8564.672529] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8564.679332] LustreError: 291476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8564.889455] Lustre: server umount lustre-MDT0001 complete [ 8572.186486] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 8573.956477] Key type lgssc unregistered [ 8574.159710] LNet: 294009:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8574.170333] LNetError: 294009:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8574.182651] LNet: Removed LNI 192.168.202.107@tcp [ 8574.723203] Key type .llcrypt unregistered [ 8574.724642] Key type ._llcrypt unregistered [ 8583.157772] Key type ._llcrypt registered [ 8583.160774] Key type .llcrypt registered [ 8583.233682] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8584.105574] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8584.134570] alg: No test for adler32 (adler32-zlib) [ 8585.122627] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 8585.304962] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 8586.952180] Key type lgssc registered [ 8587.899664] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8622.978734] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8628.180785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8628.195936] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8629.397630] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8629.417704] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8629.547498] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8632.264044] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8637.749637] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8637.837056] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8637.854979] Lustre: 296715:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8638.020133] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8640.118682] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8643.429167] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8645.641347] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8663.079476] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8663.198940] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8663.305718] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8666.715729] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8670.237976] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8671.877403] Lustre: server umount lustre-OST0000 complete [ 8676.835902] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8676.840636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8676.845207] Lustre: Skipped 1 previous similar message [ 8680.377602] Lustre: server umount lustre-MDT0000 complete [ 8681.953301] LustreError: 296728:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8681.961075] LustreError: 296728:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 8682.368946] LustreError: 296708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344011 with bad export cookie 10427230971736619123 [ 8682.370253] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8682.374096] LustreError: 296708:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8682.523171] Lustre: server umount lustre-MDT0001 complete [ 8687.825451] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 8689.831614] Key type lgssc unregistered [ 8690.071854] LNet: 299931:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8690.082488] LNetError: 299931:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8690.103937] LNet: Removed LNI 192.168.202.107@tcp [ 8690.670189] Key type .llcrypt unregistered [ 8690.671574] Key type ._llcrypt unregistered [ 8705.158090] Key type ._llcrypt registered [ 8705.159645] Key type .llcrypt registered [ 8705.229733] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8705.721889] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8705.777749] alg: No test for adler32 (adler32-zlib) [ 8706.696894] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 8706.818488] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 8708.424704] Key type lgssc registered [ 8708.980459] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8712.209687] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8712.219515] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8713.442327] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8715.302712] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8719.544230] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8719.769048] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8721.736315] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8724.412791] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8726.614629] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8732.585606] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8732.821208] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8736.023921] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8738.701422] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8749.539854] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8749.540864] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8749.546449] Lustre: Skipped 2 previous similar messages [ 8749.551477] Lustre: Skipped 2 previous similar messages [ 8750.585143] LustreError: 301306:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8750.595969] LustreError: 301306:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8750.630787] Lustre: server umount lustre-MDT0000 complete [ 8752.373749] LustreError: 303562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344081 with bad export cookie 16309792529706182501 [ 8752.375321] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8752.380359] LustreError: 303562:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8754.656945] LustreError: 301304:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8754.657234] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8754.667725] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8755.683541] LustreError: 301305:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8755.696759] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.7@tcp (stopping) [ 8758.628300] Lustre: server umount lustre-MDT0001 complete [ 8761.910430] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8766.668056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8777.184827] LustreError: 300620:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000d5a4bef3 x1866826026868480/t0(0) o250->MGC192.168.202.107@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8777.329608] LustreError: 302924:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8777.338977] LustreError: 302924:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8777.371351] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8777.398285] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:161) [ 8779.110687] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8782.499701] LustreError: 304570:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8782.508756] LustreError: 304570:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8783.814477] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8784.017139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8785.890340] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8787.113947] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8788.734261] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8792.857705] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 3 sec [ 8795.078528] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8797.029078] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 1 sec [ 8799.713648] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8799.722107] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8799.732502] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8799.735192] Lustre: Skipped 1 previous similar message [ 8802.791636] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8805.708946] Lustre: server umount lustre-OST0000 complete [ 8807.905668] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8807.907129] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8807.909810] Lustre: Skipped 1 previous similar message [ 8807.912749] Lustre: Skipped 2 previous similar messages [ 8818.145251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8818.147622] Lustre: Skipped 4 previous similar messages [ 8822.240410] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8822.353079] Lustre: server umount lustre-MDT0000 complete [ 8823.266746] LustreError: 304575:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8823.283013] LustreError: 304575:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 8824.304080] LustreError: 301291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344153 with bad export cookie 16309792529706189627 [ 8824.304492] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8824.312483] LustreError: 301291:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 8824.486193] Lustre: server umount lustre-MDT0001 complete [ 8829.607120] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 8830.939349] Key type lgssc unregistered [ 8831.079915] LNet: 306999:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8831.085776] LNetError: 306999:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8831.095740] LNet: Removed LNI 192.168.202.107@tcp [ 8831.487836] Key type .llcrypt unregistered [ 8831.492445] Key type ._llcrypt unregistered [ 8844.198904] Key type ._llcrypt registered [ 8844.200980] Key type .llcrypt registered [ 8844.249379] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8844.895566] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8844.911962] alg: No test for adler32 (adler32-zlib) [ 8845.872808] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 8846.008784] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 8847.608409] Key type lgssc registered [ 8848.126506] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8851.555643] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8851.564160] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8852.842377] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8854.697720] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8858.534355] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8858.695108] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8860.402842] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8862.785959] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8864.296773] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8868.321047] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8868.322312] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8868.327793] Lustre: Skipped 1 previous similar message [ 8868.330915] Lustre: Skipped 1 previous similar message [ 8869.345274] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8869.347605] Lustre: Skipped 1 previous similar message [ 8871.604665] Lustre: server umount lustre-MDT0000 complete [ 8873.228860] LustreError: 308356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344202 with bad export cookie 12617642614120118052 [ 8873.232570] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8873.236828] LustreError: 308356:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8873.375824] Lustre: server umount lustre-MDT0001 complete [ 8879.286280] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8879.431886] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8879.444526] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8879.534256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8881.285179] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8885.113196] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8885.154167] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8885.170353] Lustre: 310735:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8885.292375] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8886.986805] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8889.452125] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8891.026956] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8896.355640] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8896.432930] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8896.437920] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8896.442922] LustreError: 312391:0:(tgt_mount.c:2569:server_fill_super()) Unable to start targets: -2 [ 8896.446673] LustreError: 312391:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 8896.450482] LustreError: 312391:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8896.476363] Lustre: server umount lustre-OST0000 complete [ 8896.477737] LustreError: 312391:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8901.731705] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8901.808784] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8901.884665] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8904.151926] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8906.430392] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8910.044326] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8910.079544] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8910.116100] Lustre: lustre-OST0001: new disk, initializing [ 8910.118705] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8910.122337] Lustre: 313809:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8910.173875] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8911.211756] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8911.216694] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8911.242291] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:134 to 0x280000401:193) [ 8911.291933] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8912.656917] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 8915.130651] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 8916.451914] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8916.453140] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8916.456317] LustreError: Skipped 1 previous similar message [ 8916.457508] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8916.462698] Lustre: Skipped 1 previous similar message [ 8921.571188] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8921.577541] Lustre: Skipped 2 previous similar messages [ 8922.436458] Lustre: server umount lustre-OST0001 complete [ 8926.693202] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8926.693336] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8926.694909] LustreError: 313952:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8926.694915] LustreError: 313952:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8926.696465] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8926.719047] Lustre: Skipped 1 previous similar message [ 8930.091436] Lustre: server umount lustre-OST0000 complete [ 8931.809153] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8936.418210] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8936.422141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8936.427329] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8936.433586] Lustre: Skipped 2 previous similar messages [ 8946.144131] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8946.221112] Lustre: server umount lustre-MDT0000 complete [ 8947.657286] LustreError: 311457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344276 with bad export cookie 12617642614120118591 [ 8947.659239] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8947.662569] LustreError: 311457:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8947.775677] Lustre: server umount lustre-MDT0001 complete [ 8951.781623] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 8952.932568] Key type lgssc unregistered [ 8953.069229] LNet: 315641:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8953.073845] LNetError: 315641:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8953.084781] LNet: Removed LNI 192.168.202.107@tcp [ 8953.408233] Key type .llcrypt unregistered [ 8953.409422] Key type ._llcrypt unregistered [ 8988.814698] Key type ._llcrypt registered [ 8988.815958] Key type .llcrypt registered [ 8988.866654] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 8996.215869] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 8996.600655] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8996.641372] alg: No test for adler32 (adler32-zlib) [ 8997.511693] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 8997.605964] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 8999.200173] Key type lgssc registered [ 8999.726970] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9015.302660] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9015.309834] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9016.422330] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9016.437192] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9016.477783] Lustre: lustre-MDT0000: new disk, initializing [ 9016.513962] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9016.523744] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9018.157872] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9022.743112] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9022.783159] Lustre: 320512:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9022.804867] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9022.807211] Lustre: Skipped 1 previous similar message [ 9022.849147] Lustre: lustre-MDT0001: new disk, initializing [ 9022.886166] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9022.898492] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9022.901709] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9024.448599] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9027.971387] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9029.508591] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9032.287568] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9032.386089] Lustre: lustre-OST0000: new disk, initializing [ 9032.388486] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9032.392323] Lustre: 322255:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9032.428380] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9034.466610] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9037.863408] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9038.847924] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9038.852795] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9038.891616] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9039.440225] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9039.519413] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9040.990688] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9041.064743] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9043.937665] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9043.937821] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9043.940569] LustreError: Skipped 1 previous similar message [ 9043.941048] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9043.945045] Lustre: Skipped 1 previous similar message [ 9047.906628] Lustre: server umount lustre-OST0000 complete [ 9053.664897] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9053.667453] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9053.672281] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9053.674344] Lustre: Skipped 1 previous similar message [ 9055.456617] Lustre: server umount lustre-MDT0000 complete [ 9056.908182] LustreError: 320504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344385 with bad export cookie 4664550194668801219 [ 9056.910996] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9056.912022] LustreError: 320504:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9057.040986] Lustre: server umount lustre-MDT0001 complete [ 9060.371471] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 16:06:29 (1780344389) [ 9063.182929] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 16:06:32 (1780344392) [ 9066.379065] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9066.625081] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9068.229266] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9072.043790] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9072.097673] LustreError: 324858:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9072.108653] LustreError: 324858:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9072.225416] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9074.001739] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9076.135985] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9077.618557] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9080.264840] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9080.393357] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9082.644366] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9084.794597] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9099.683776] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 9100.267691] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9100.884103] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 9101.550505] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 9102.384242] Lustre: Failing over lustre-MDT0000 [ 9102.529760] Lustre: server umount lustre-MDT0000 complete [ 9102.816788] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9102.820636] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9102.824754] Lustre: Skipped 1 previous similar message [ 9102.827835] LustreError: 324876:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9106.406854] LustreError: 324852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9107.427065] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9107.431985] Lustre: Skipped 1 previous similar message [ 9111.519546] LustreError: 324852:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.202.7@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9111.526801] LustreError: 324852:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9115.821548] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9115.890307] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9116.005928] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9116.035078] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9116.640810] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9117.616626] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9121.253780] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9121.268717] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 9121.285763] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 9122.726970] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9123.362052] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9125.323948] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 9126.028227] Lustre: Failing over lustre-MDT0000 [ 9126.185659] Lustre: server umount lustre-MDT0000 complete [ 9126.369765] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9126.371315] LustreError: 324853:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9126.372118] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9126.374150] Lustre: Skipped 2 previous similar messages [ 9126.380128] LustreError: 324853:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 9136.610172] LustreError: 324857:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9136.623755] LustreError: 324857:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 9139.852340] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9139.905102] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9140.033300] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 9141.426635] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9142.240052] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 9145.318799] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 9145.320910] Lustre: Skipped 2 previous similar messages [ 9145.329069] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 9145.349842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 9146.793647] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9147.491631] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9150.434442] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9150.434596] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9150.437035] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9150.437043] Lustre: Skipped 2 previous similar messages [ 9150.441799] Lustre: Skipped 2 previous similar messages [ 9155.553307] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9155.558600] Lustre: Skipped 4 previous similar messages [ 9156.078409] Lustre: server umount lustre-MDT0000 complete [ 9157.835936] LustreError: 324838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344486 with bad export cookie 4664550194668803942 [ 9157.840436] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9157.842182] LustreError: 324838:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9157.984908] Lustre: server umount lustre-MDT0001 complete [ 9169.890521] Lustre: server umount lustre-OST0000 complete [ 9176.535754] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9176.689187] LustreError: 330643:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9176.696282] LustreError: 330643:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9176.715557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9176.718674] Lustre: Skipped 1 previous similar message [ 9178.222598] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9181.539403] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9183.181535] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9185.197980] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9186.645903] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9189.279097] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9191.595800] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9193.746738] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9202.726063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9206.753094] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9206.756514] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9206.761960] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9211.675462] Lustre: server umount lustre-OST0000 complete [ 9217.506091] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9217.515355] Lustre: Skipped 2 previous similar messages [ 9219.247918] Lustre: server umount lustre-MDT0000 complete [ 9220.874305] LustreError: 332802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344549 with bad export cookie 4664550194668804663 [ 9220.875653] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9220.878207] LustreError: 332802:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9221.046374] Lustre: server umount lustre-MDT0001 complete [ 9225.675209] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 9226.941376] Key type lgssc unregistered [ 9227.098652] LNet: 333837:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9227.102215] LNetError: 333837:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9227.111620] LNet: Removed LNI 192.168.202.107@tcp [ 9227.480498] Key type .llcrypt unregistered [ 9227.482186] Key type ._llcrypt unregistered [ 9238.646818] Key type ._llcrypt registered [ 9238.648367] Key type .llcrypt registered [ 9238.692913] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9239.139194] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9239.157517] alg: No test for adler32 (adler32-zlib) [ 9240.022867] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 9240.115731] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 9241.696145] Key type lgssc registered [ 9242.084084] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9244.843902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9244.851335] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9246.040844] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9247.465143] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9250.419904] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9250.556249] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9251.991129] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9253.896666] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9255.220785] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9257.788610] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9257.911200] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9260.036824] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9262.156128] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9271.211821] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9281.506852] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9281.508681] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9281.510195] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9281.526290] Lustre: Skipped 1 previous similar message [ 9283.164125] Lustre: server umount lustre-MDT0000 complete [ 9284.744345] LustreError: 335301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344613 with bad export cookie 18124218751039454128 [ 9284.746472] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9284.753530] LustreError: 335301:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9284.909268] Lustre: server umount lustre-MDT0001 complete [ 9296.721550] Lustre: server umount lustre-OST0000 complete [ 9303.395507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9303.644605] LustreError: 338710:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9303.652757] LustreError: 338710:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9303.687569] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9305.508788] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9309.159025] LustreError: 338711:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9310.004634] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9310.180563] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9312.215327] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9315.088095] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9316.733600] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9319.832618] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9320.002164] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9322.558082] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9325.160799] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9328.231595] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9338.853649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9338.855387] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9338.858060] Lustre: Skipped 3 previous similar messages [ 9338.862739] Lustre: Skipped 4 previous similar messages [ 9343.649772] Lustre: server umount lustre-MDT0000 complete [ 9343.970472] LustreError: 339422:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9343.981669] LustreError: 339422:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9345.040345] LustreError: 338690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344674 with bad export cookie 18124218751039455227 [ 9345.040792] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9345.045712] LustreError: 338690:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 9345.185572] Lustre: server umount lustre-MDT0001 complete [ 9356.952118] Lustre: server umount lustre-OST0000 complete [ 9361.992812] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9362.129324] LustreError: 342279:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9362.149878] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9363.336168] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9366.276776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9367.908490] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9369.766664] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9371.006738] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9373.585064] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9375.825484] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9377.721269] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9387.045771] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9392.097081] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9392.100404] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9392.101364] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9392.105774] LustreError: Skipped 1 previous similar message [ 9395.853993] Lustre: server umount lustre-OST0000 complete [ 9397.217718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9397.217854] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9397.222077] Lustre: Skipped 1 previous similar message [ 9397.227875] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9397.230778] Lustre: Skipped 2 previous similar messages [ 9402.337240] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9402.340589] Lustre: Skipped 1 previous similar message [ 9403.289960] Lustre: server umount lustre-MDT0000 complete [ 9404.695343] LustreError: 342261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344733 with bad export cookie 18124218751039456235 [ 9404.698309] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9404.699816] LustreError: 342261:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9404.820398] Lustre: server umount lustre-MDT0001 complete [ 9408.625043] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 9409.707399] Key type lgssc unregistered [ 9409.839298] LNet: 345472:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9409.843571] LNetError: 345472:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9409.852344] LNet: Removed LNI 192.168.202.107@tcp [ 9410.155519] Key type .llcrypt unregistered [ 9410.156695] Key type ._llcrypt unregistered [ 9420.372778] Key type ._llcrypt registered [ 9420.374147] Key type .llcrypt registered [ 9420.418319] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9420.784621] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9420.818022] alg: No test for adler32 (adler32-zlib) [ 9421.671031] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 9421.747552] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 9423.328149] Key type lgssc registered [ 9423.713771] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9426.518311] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9426.524775] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9426.528726] LustreError: 346872:0:(osd_handler.c:8553:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9426.538297] LustreError: 346872:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9426.542083] LustreError: 346872:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9426.545754] LustreError: 346872:0:(tgt_mount.c:2537:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9426.550104] LustreError: 346872:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9429.489187] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9429.491798] LustreError: 347269:0:(osd_handler.c:8553:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9429.500202] LustreError: 347269:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9429.503025] LustreError: 347269:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9429.505592] LustreError: 347269:0:(tgt_mount.c:2537:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9429.508161] LustreError: 347269:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9436.242940] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 9437.327632] Key type lgssc unregistered [ 9437.471995] LNet: 348252:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9437.476821] LNetError: 348252:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9437.489509] LNet: Removed LNI 192.168.202.107@tcp [ 9437.815527] Key type .llcrypt unregistered [ 9437.817385] Key type ._llcrypt unregistered [ 9444.739353] Key type ._llcrypt registered [ 9444.740915] Key type .llcrypt registered [ 9444.780648] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 9451.813068] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9452.112964] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9452.122467] alg: No test for adler32 (adler32-zlib) [ 9453.013924] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 9453.118453] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 9454.712166] Key type lgssc registered [ 9455.238667] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9469.452924] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9469.461168] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9470.602662] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9470.621899] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9470.664460] Lustre: lustre-MDT0000: new disk, initializing [ 9470.706143] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9470.716997] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9472.509258] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9478.337553] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9478.391784] Lustre: 351348:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9478.428168] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9478.431390] Lustre: Skipped 1 previous similar message [ 9478.476943] Lustre: lustre-MDT0001: new disk, initializing [ 9478.522876] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9478.536865] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9478.540104] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9480.523650] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9484.341758] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9486.071255] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9489.180979] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9489.302546] Lustre: lustre-OST0000: new disk, initializing [ 9489.304625] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9489.308127] Lustre: 353092:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9489.347178] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9491.656507] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9491.659489] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9491.733943] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9491.751597] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9495.507054] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9497.023708] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9497.122376] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9499.025115] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9499.148196] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9502.179518] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9502.181297] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9502.186521] Lustre: Skipped 1 previous similar message [ 9502.188510] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9502.192171] Lustre: Skipped 1 previous similar message [ 9506.440519] Lustre: server umount lustre-OST0000 complete [ 9509.346158] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9509.349171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9509.355149] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9512.417503] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9512.418412] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9514.666749] Lustre: server umount lustre-MDT0000 complete [ 9516.280991] LustreError: 353091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344845 with bad export cookie 15443910655166539402 [ 9516.283700] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9516.285744] LustreError: 353091:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9516.428355] Lustre: server umount lustre-MDT0001 complete [ 9520.717543] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9521.701666] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 16:14:10 (1780344850) [ 9525.141206] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9525.436103] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9527.470932] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9530.849418] LustreError: 355400:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9530.858508] LustreError: 355400:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 9531.331499] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9531.503031] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9533.408926] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9536.172983] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9538.045356] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9541.490584] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9541.673222] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9544.633885] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9547.491825] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9551.266153] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9551.341320] Lustre: lustre-OST0001: new disk, initializing [ 9551.344180] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9551.347531] Lustre: 357895:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9554.482381] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9558.041099] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9558.045099] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9558.085729] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9558.350550] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 9563.109947] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9563.110827] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9563.116081] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9563.120200] Lustre: Skipped 1 previous similar message [ 9563.121991] Lustre: Skipped 1 previous similar message [ 9566.526575] Lustre: server umount lustre-OST0001 complete [ 9568.227564] LustreError: 358459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9568.242281] LustreError: 358459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9573.352254] LustreError: 356997:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9573.360094] LustreError: 356997:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9578.468427] LustreError: 358459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9578.475074] LustreError: 358459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9581.537121] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9581.539790] LustreError: Skipped 1 previous similar message [ 9581.545147] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9581.550356] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9581.553840] Lustre: Skipped 2 previous similar messages [ 9583.588500] LustreError: 358459:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9583.594533] LustreError: 358459:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9593.312120] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9593.372751] Lustre: server umount lustre-OST0000 complete [ 9597.920939] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9597.924488] LustreError: Skipped 1 previous similar message [ 9597.926460] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9597.930833] Lustre: Skipped 1 previous similar message [ 9597.932746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9597.934904] Lustre: Skipped 4 previous similar messages [ 9601.129763] Lustre: server umount lustre-MDT0000 complete [ 9602.951707] LustreError: 357894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780344931 with bad export cookie 15443910655166540333 [ 9602.953272] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9602.959103] LustreError: 357894:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9617.376109] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9617.472625] Lustre: server umount lustre-MDT0001 complete [ 9621.756372] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 9622.925393] Key type lgssc unregistered [ 9623.077545] LNet: 359753:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9623.081854] LNetError: 359753:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9623.092487] LNet: Removed LNI 192.168.202.107@tcp [ 9623.458307] Key type .llcrypt unregistered [ 9623.461264] Key type ._llcrypt unregistered [ 9646.924558] Key type ._llcrypt registered [ 9646.927402] Key type .llcrypt registered [ 9646.977489] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9647.346424] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9647.379224] alg: No test for adler32 (adler32-zlib) [ 9648.252279] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 9648.342210] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 9649.936163] Key type lgssc registered [ 9650.399523] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9653.379470] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9653.386345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9654.477508] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9654.486381] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9654.544702] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9655.841126] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9658.797391] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9658.835303] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9658.847554] Lustre: 362510:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9658.941441] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9660.240050] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9662.323770] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9663.746035] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9666.019704] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9666.020918] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9666.025761] Lustre: Skipped 1 previous similar message [ 9666.030972] Lustre: Skipped 1 previous similar message [ 9670.689521] Lustre: server umount lustre-MDT0000 complete [ 9672.329152] LustreError: 362501:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345001 with bad export cookie 11012624732505859900 [ 9672.335461] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9672.490455] Lustre: server umount lustre-MDT0001 complete [ 9676.630928] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 16:16:45 (1780345005) [ 9679.320211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9679.539971] LustreError: 364794:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9679.545794] LustreError: 364794:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9679.558754] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9681.085698] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9684.912023] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9684.963273] LustreError: 364795:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9685.119599] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9686.760424] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9689.314958] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9690.947946] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9693.784527] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9693.887600] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 9693.977484] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9696.446117] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9698.847435] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9704.002594] Lustre: Permanently deactivating lustre-OST0000 [ 9704.009550] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 9704.353363] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9704.733556] LustreError: 367248:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 9704.737816] LustreError: 367248:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 9704.740468] LustreError: 367248:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9705.373688] LustreError: 367296:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 9705.376374] LustreError: 367296:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9712.160870] Lustre: server umount lustre-OST0000 complete [ 9713.426042] LustreError: 367545:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 9713.428405] LustreError: 367545:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 9716.192858] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9716.195765] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9716.200502] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9718.754671] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9718.754703] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9718.763560] Lustre: Skipped 1 previous similar message [ 9723.873403] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9723.876040] Lustre: Skipped 1 previous similar message [ 9724.930737] Lustre: server umount lustre-MDT0000 complete [ 9726.349276] LustreError: 364776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345055 with bad export cookie 11012624732505860467 [ 9726.352145] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9726.354445] LustreError: 364776:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 8 previous similar messages [ 9736.743581] Lustre: server umount lustre-MDT0001 complete [ 9740.769232] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9742.268386] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9743.253505] LustreError: 368684:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 9745.384099] Lustre: 369024:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.202.107@tcp [ 9746.137811] Lustre: server umount MGS complete [ 9750.161215] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9750.326535] LustreError: 369746:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9750.337260] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9750.340084] Lustre: Skipped 1 previous similar message [ 9750.361164] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9751.877144] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9755.402609] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9755.521500] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 9757.275093] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9759.996335] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9761.991592] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9762.880416] Lustre: Permanently reactivating lustre-OST0000 [ 9762.893972] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 9762.903883] Lustre: Skipped 2 previous similar messages [ 9789.410234] LustreError: 371309:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 9793.102352] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9793.367743] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9793.373192] Lustre: Skipped 1 previous similar message [ 9796.330248] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9798.906629] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9813.985521] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9813.989663] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9813.995996] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9815.013075] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9816.990930] Lustre: server umount lustre-OST0000 complete [ 9820.129539] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9820.133357] Lustre: Skipped 1 previous similar message [ 9824.532259] Lustre: server umount lustre-MDT0000 complete [ 9825.250085] LustreError: 369747:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9825.254251] LustreError: 369747:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 9825.932051] LustreError: 371096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345154 with bad export cookie 11012624732505861503 [ 9825.934875] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9825.937178] LustreError: 371096:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9826.046995] Lustre: server umount lustre-MDT0001 complete [ 9830.477327] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 9831.572346] Key type lgssc unregistered [ 9831.708549] LNet: 374013:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9831.711556] LNetError: 374013:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9831.723400] LNet: Removed LNI 192.168.202.107@tcp [ 9832.050310] Key type .llcrypt unregistered [ 9832.051613] Key type ._llcrypt unregistered [ 9838.773379] Key type ._llcrypt registered [ 9838.775106] Key type .llcrypt registered [ 9838.814524] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [ 9846.061844] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9846.588164] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9846.602063] alg: No test for adler32 (adler32-zlib) [ 9847.500555] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 9847.616347] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 9849.232144] Key type lgssc registered [ 9849.724827] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9861.825454] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9861.833898] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9862.923764] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9862.938066] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9862.979776] Lustre: lustre-MDT0000: new disk, initializing [ 9863.010496] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9863.018390] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9864.590379] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9869.383135] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9869.438859] Lustre: 377110:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9869.475573] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9869.478228] Lustre: Skipped 1 previous similar message [ 9869.518550] Lustre: lustre-MDT0001: new disk, initializing [ 9869.561793] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9869.579828] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9869.587796] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9871.153393] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9874.877643] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9876.411326] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9879.255814] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9879.373590] Lustre: lustre-OST0000: new disk, initializing [ 9879.376188] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9879.381124] Lustre: 378853:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9879.419130] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9881.611481] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9885.048972] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9886.611178] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9886.724670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9886.728901] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9886.766935] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9887.717881] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 9889.293699] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9889.375690] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9891.809852] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9891.811540] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9891.812416] LustreError: Skipped 1 previous similar message [ 9891.818700] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9891.822100] Lustre: Skipped 1 previous similar message [ 9896.224192] Lustre: server umount lustre-OST0000 complete [ 9900.512869] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9900.516118] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9900.519624] Lustre: Skipped 1 previous similar message [ 9900.521334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9902.049257] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9902.049394] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9907.169055] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9907.170978] Lustre: Skipped 2 previous similar messages [ 9911.266477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9911.269242] Lustre: Skipped 1 previous similar message [ 9912.289045] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9912.353871] Lustre: server umount lustre-MDT0000 complete [ 9913.821144] LustreError: 377104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345242 with bad export cookie 7754939690018274632 [ 9913.824038] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9913.828299] LustreError: 377104:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9913.946554] Lustre: server umount lustre-MDT0001 complete [ 9917.069137] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 9917.734397] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 16:20:46 (1780345246) [ 9920.738980] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9921.006845] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9922.878626] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9926.114946] LustreError: 381170:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9926.133568] LustreError: 381170:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 9926.895657] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9927.100406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9928.852485] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9931.085280] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 9932.875035] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 9935.557336] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9935.722290] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9938.184590] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [ 9940.711928] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 9941.650880] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 9947.720054] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 9951.203552] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9951.204081] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9951.204863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9951.215244] Lustre: Skipped 1 previous similar message [ 9955.412458] Lustre: server umount lustre-OST0000 complete [ 9957.856828] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9957.860054] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9963.292394] Lustre: server umount lustre-MDT0000 complete [ 9964.742969] LustreError: 382788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345293 with bad export cookie 7754939690018275556 [ 9964.745944] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9964.748880] LustreError: 382788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9964.890185] Lustre: server umount lustre-MDT0001 complete [ 9969.532855] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 9970.980567] Key type lgssc unregistered [ 9971.189092] LNet: 384542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9971.200680] LNetError: 384542:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9971.213767] LNet: Removed LNI 192.168.202.107@tcp [ 9971.711484] Key type .llcrypt unregistered [ 9971.713176] Key type ._llcrypt unregistered [ 9982.193285] Key type ._llcrypt registered [ 9982.195014] Key type .llcrypt registered [ 9982.257290] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [ 9982.406241] Key type .llcrypt unregistered [ 9982.407251] Key type ._llcrypt unregistered [ 9991.262264] Key type ._llcrypt registered [ 9991.263787] Key type .llcrypt registered [ 9991.320983] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [ 9991.730203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9991.780888] alg: No test for adler32 (adler32-zlib) [ 9992.698752] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [ 9992.805836] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [ 9994.394400] Key type lgssc registered [ 9994.895724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9998.085416] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9998.092241] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9999.288396] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10001.064525] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10005.134111] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10005.335634] LustreError: 387424:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10005.348415] LustreError: 387424:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10005.349631] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10008.020199] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10009.443276] LustreError: 386698:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10009.447852] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10009.450520] LustreError: 386698:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10010.598884] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10013.348115] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10013.503328] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10015.296516] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10023.907047] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10023.907533] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10023.915159] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10023.920952] Lustre: Skipped 1 previous similar message [10028.908586] Lustre: server umount lustre-OST0000 complete [10033.125805] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10033.128878] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10033.132881] Lustre: Skipped 1 previous similar message [10033.134650] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10034.145185] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10034.145604] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10036.790540] Lustre: server umount lustre-MDT0000 complete [10038.552971] LustreError: 386678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345367 with bad export cookie 17987940176942889553 [10038.555693] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10038.557238] LustreError: 386678:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10038.727934] Lustre: server umount lustre-MDT0001 complete [10043.193951] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [10044.444615] Key type lgssc unregistered [10044.619706] LNet: 389680:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10044.623775] LNetError: 389680:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10045.670343] LNet: Removed LNI 192.168.202.107@tcp [10045.977279] Key type .llcrypt unregistered [10045.978462] Key type ._llcrypt unregistered [10056.181016] Key type ._llcrypt registered [10056.182108] Key type .llcrypt registered [10056.219763] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10056.638502] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10056.648631] alg: No test for adler32 (adler32-zlib) [10057.522249] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [10057.626160] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [10059.224193] Key type lgssc registered [10059.677527] Lustre: Echo OBD driver; http://www.lustre.org/ [10062.748095] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10062.757660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10078.240278] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10083.360139] LustreError: 391030:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.202.107@tcp: failed processing log, type 4: rc = -110 [10109.984216] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10112.057070] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10114.225961] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10116.949360] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10117.161495] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10117.184046] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10118.657467] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10121.730045] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10121.864754] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10123.768892] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10126.309551] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10127.945234] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10132.450540] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10132.450680] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10132.453688] LustreError: Skipped 1 previous similar message [10132.459385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10132.463293] Lustre: Skipped 1 previous similar message [10135.522233] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10135.525066] Lustre: Skipped 1 previous similar message [10136.657821] Lustre: server umount lustre-OST0000 complete [10140.643482] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10140.644412] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10140.647540] Lustre: Skipped 2 previous similar messages [10144.149976] Lustre: server umount lustre-MDT0000 complete [10145.581286] LustreError: 391037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345474 with bad export cookie 10753013201466832365 [10145.584174] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10145.586042] LustreError: 391037:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10145.708098] Lustre: server umount lustre-MDT0001 complete [10149.648138] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [10150.798576] Key type lgssc unregistered [10150.939399] LNet: 394241:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10150.943527] LNetError: 394241:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10150.954588] LNet: Removed LNI 192.168.202.107@tcp [10151.306454] Key type .llcrypt unregistered [10151.308175] Key type ._llcrypt unregistered [10161.114813] Key type ._llcrypt registered [10161.115923] Key type .llcrypt registered [10161.153779] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10161.573257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10161.581254] alg: No test for adler32 (adler32-zlib) [10162.438995] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [10162.528835] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [10164.112141] Key type lgssc registered [10164.540460] Lustre: Echo OBD driver; http://www.lustre.org/ [10167.350903] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10167.358051] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10168.535252] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10170.052910] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10173.186965] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10173.335600] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10174.682941] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10176.720694] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10178.093727] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10180.628236] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10180.771859] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10182.913759] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10184.927463] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10193.063518] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10198.498257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10198.499509] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10198.506145] Lustre: Skipped 2 previous similar messages [10198.508377] Lustre: Skipped 1 previous similar message [10200.549466] Lustre: lustre-MDT0000: Not available for connect from 192.168.202.7@tcp (stopping) [10200.551349] Lustre: Skipped 1 previous similar message [10202.586631] Lustre: server umount lustre-MDT0000 complete [10203.363315] LustreError: 395599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.202.7@tcp arrived at 1780345532 with bad export cookie 5450969088398276727 [10203.368620] LustreError: 395599:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10203.616906] LustreError: 395641:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10203.622973] LustreError: 395641:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [10204.062244] LustreError: 395598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345533 with bad export cookie 5450969088398276104 [10204.066432] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10204.066687] LustreError: 395598:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10204.640967] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10204.643719] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10204.648981] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10208.736870] LustreError: 395617:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10208.737172] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10208.744299] LustreError: 395617:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10209.760987] LustreError: 395616:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10209.769553] LustreError: 395616:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10210.204969] Lustre: server umount lustre-OST0000 complete [10212.687603] Lustre: server umount lustre-MDT0001 complete [10216.601381] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [10217.674952] Key type lgssc unregistered [10217.804430] LNet: 398965:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10217.807232] LNetError: 398965:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10217.815362] LNet: Removed LNI 192.168.202.107@tcp [10218.138644] Key type .llcrypt unregistered [10218.139899] Key type ._llcrypt unregistered [10228.591801] Key type ._llcrypt registered [10228.592909] Key type .llcrypt registered [10228.640525] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10229.060904] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10229.104488] alg: No test for adler32 (adler32-zlib) [10229.983722] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [10230.093373] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [10231.696142] Key type lgssc registered [10232.116692] Lustre: Echo OBD driver; http://www.lustre.org/ [10235.250196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10235.257055] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10236.449220] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10237.947194] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10240.890843] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10241.028061] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10242.342541] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10244.360756] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10245.665813] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10248.012722] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10248.135131] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10250.112454] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10251.977713] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10262.505280] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [10266.592890] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10266.596449] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10266.601952] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10267.626319] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10267.627214] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10267.638191] Lustre: Skipped 1 previous similar message [10268.653309] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.7@tcp (stopping) [10270.893706] Lustre: server umount lustre-MDT0001 complete [10272.740417] LustreError: 400344:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10272.753519] LustreError: 400344:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10277.857336] LustreError: 400343:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10277.857745] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10277.869178] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10277.876433] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10279.195275] Lustre: server umount lustre-OST0000 complete [10280.728316] Lustre: server umount lustre-MDT0000 complete [10285.834564] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [10286.916912] Key type lgssc unregistered [10287.044444] LNet: 403693:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10287.047989] LNetError: 403693:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10287.057346] LNet: Removed LNI 192.168.202.107@tcp [10287.380880] Key type .llcrypt unregistered [10287.382070] Key type ._llcrypt unregistered [10296.603251] Key type ._llcrypt registered [10296.604679] Key type .llcrypt registered [10296.659106] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [10303.478248] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10303.817985] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10303.832980] alg: No test for adler32 (adler32-zlib) [10304.714510] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [10304.819304] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [10306.408125] Key type lgssc registered [10306.919962] Lustre: Echo OBD driver; http://www.lustre.org/ [10327.774248] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10335.141196] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10335.155395] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10336.275300] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10336.291458] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10336.330509] Lustre: lustre-MDT0000: new disk, initializing [10336.378791] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10336.386957] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10338.142851] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10344.932960] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10345.030357] Lustre: lustre-OST0000: new disk, initializing [10345.032029] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10345.033727] Lustre: Skipped 1 previous similar message [10345.035780] Lustre: 408647:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10345.068709] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10347.442189] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10351.142991] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10352.644832] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [10352.648667] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [10352.659960] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [10363.714716] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10363.778685] Lustre: 407647:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10363.823931] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10363.867240] Lustre: lustre-MDT0001: new disk, initializing [10363.937076] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10363.959712] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [10363.967184] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [10366.900987] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10377.883459] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [10378.315961] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [10383.335990] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [10383.582123] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [10391.603750] Lustre: lustre-MDT0001: Not available for connect from 192.168.202.7@tcp (stopping) [10393.069150] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10393.072222] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [10393.093190] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [10393.147925] Lustre: Skipped 1 previous similar message [10395.906370] Lustre: server umount lustre-MDT0001 complete [10403.306803] LustreError: 407652:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10403.316180] LustreError: 407652:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [10408.439921] LustreError: 407654:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10408.480810] LustreError: 407654:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10413.564166] LustreError: 407658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10413.576165] LustreError: 407658:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10414.561359] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10414.592090] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10414.599045] Lustre: Skipped 1 previous similar message [10414.601830] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10418.657898] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10418.659507] LustreError: 409861:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10420.342666] Lustre: server umount lustre-OST0000 complete [10423.795658] LustreError: 407658:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10430.770277] Lustre: server umount lustre-MDT0000 complete [10446.734725] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [10450.556908] Key type lgssc unregistered [10450.848229] LNet: 412214:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10450.863643] LNetError: 412214:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [10450.895865] LNet: Removed LNI 192.168.202.107@tcp [10452.104126] Key type .llcrypt unregistered [10452.106985] Key type ._llcrypt unregistered [10488.474834] Key type ._llcrypt registered [10488.476535] Key type .llcrypt registered [10488.557631] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [10488.788243] Key type .llcrypt unregistered [10488.789628] Key type ._llcrypt unregistered [10501.648143] Key type ._llcrypt registered [10501.650277] Key type .llcrypt registered [10501.745743] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [10502.523707] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10502.597550] alg: No test for adler32 (adler32-zlib) [10503.605135] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [10503.788310] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [10505.432259] Key type lgssc registered [10506.286628] Lustre: Echo OBD driver; http://www.lustre.org/ [10528.629415] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10528.652964] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10529.883441] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10529.902410] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10529.977930] Lustre: lustre-MDT0000: new disk, initializing [10530.074233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10530.092444] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10533.278724] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10541.481886] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10541.563791] Lustre: 415535:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10541.611891] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10541.615663] Lustre: Skipped 1 previous similar message [10541.685498] Lustre: lustre-MDT0001: new disk, initializing [10541.781400] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10541.814793] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10541.827499] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10545.554594] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10551.550892] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10554.130656] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10559.370268] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10559.631806] Lustre: lustre-OST0000: new disk, initializing [10559.641452] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10559.649745] Lustre: 417279:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10559.715192] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10564.455843] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10569.766238] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10569.775438] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10569.837260] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10570.298251] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10574.818369] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10574.821602] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10574.838162] Lustre: Skipped 2 previous similar messages [10574.840261] Lustre: Skipped 1 previous similar message [10579.937683] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10579.943054] Lustre: Skipped 3 previous similar messages [10580.698927] Lustre: server umount lustre-MDT0000 complete [10583.182754] LustreError: 416324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780345912 with bad export cookie 8353168923440005395 [10583.184018] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10583.198305] LustreError: 416324:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10583.361597] Lustre: server umount lustre-MDT0001 complete [10596.073764] Lustre: server umount lustre-OST0000 complete [10604.838326] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 16:32:12 (1780345932) [10611.180284] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10611.651839] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10615.203578] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10616.811718] LustreError: 419192:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10616.834258] LustreError: 419192:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [10621.922699] LustreError: 419191:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10621.966703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10622.307266] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10626.229685] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10630.711233] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [10633.688853] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [10652.013808] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10652.238810] Lustre: Found index 0 for lustre-OST0000, updating log [10652.424856] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10656.567547] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [10657.078187] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [10660.629772] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [10666.979419] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [10666.987819] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10666.994787] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10669.096567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10669.102086] Lustre: Skipped 1 previous similar message [10669.877325] Lustre: server umount lustre-OST0000 complete [10685.410049] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10685.410765] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10685.429026] Lustre: Skipped 1 previous similar message [10690.440602] Lustre: server umount lustre-MDT0000 complete [10690.532151] LustreError: 421446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10690.559445] LustreError: 421446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [10692.578945] LustreError: 421446:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10692.591438] LustreError: 421446:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [10693.278658] LustreError: 419171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346022 with bad export cookie 8353168923440006830 [10693.284164] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10693.290300] LustreError: 419171:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10693.507954] Lustre: server umount lustre-MDT0001 complete [10699.023368] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 16:33:47 (1780346027) [10716.768263] LNet: 1 local NIs in recovery (showing 1): 192.168.202.107@tcp [11298.962918] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11299.651792] LustreError: 425259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11299.762906] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11304.353624] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11310.061141] LustreError: 425260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11310.075220] LustreError: 425260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [11312.858298] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11313.313634] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11317.622515] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11323.105615] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11326.609894] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11346.059546] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11346.283857] Lustre: Found index 0 for lustre-OST0000, updating log [11346.536690] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11349.616794] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [11353.359816] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11355.745893] LNet: 1 local NIs in recovery (showing 1): 192.168.202.107@tcp [11358.639261] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11362.153767] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11362.165183] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11362.169091] Lustre: Skipped 1 previous similar message [11367.433803] Lustre: server umount lustre-OST0000 complete [11372.581103] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11372.584262] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11372.599646] Lustre: Skipped 1 previous similar message [11372.611859] Lustre: Skipped 1 previous similar message [11374.709110] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11374.720259] Lustre: Skipped 2 previous similar messages [11377.958894] Lustre: server umount lustre-MDT0000 complete [11378.786884] LustreError: 425259:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11378.818388] LustreError: 425259:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [11381.829290] LustreError: 425239:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780346710 with bad export cookie 8353168923440007901 [11381.836181] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11382.266653] Lustre: server umount lustre-MDT0001 complete [11412.517746] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [11419.233486] LNet: 1 local NIs in recovery (showing 1): 192.168.202.107@tcp [11423.715930] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [11460.661974] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 16:46:28 (1780346788) [11480.780137] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [11483.449914] Key type lgssc unregistered [11483.704431] LNet: 431477:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11483.715566] LNetError: 431477:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11609.709545] LNet: Removed LNI 192.168.202.107@tcp [11610.638609] Key type .llcrypt unregistered [11610.640913] Key type ._llcrypt unregistered [11628.240559] Key type ._llcrypt registered [11628.243742] Key type .llcrypt registered [11628.334806] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [11630.111289] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11630.157641] alg: No test for adler32 (adler32-zlib) [11631.442567] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [11631.682436] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [11633.365305] Key type lgssc registered [11634.675982] Lustre: Echo OBD driver; http://www.lustre.org/ [11639.904695] LNet: Added LNI 192.168.202.107@tcp100 [8/256/0/180] [11640.925484] LNet: Added LNI 192.168.202.107@tcp101 [8/256/0/180] [11642.242465] LNet: Added LNI 192.168.202.107@tcp102 [8/256/0/180] [11643.383168] LNet: Added LNI 192.168.202.107@tcp103 [8/256/0/180] [11644.376207] LNet: Added LNI 192.168.202.107@tcp104 [8/256/0/180] [11645.338471] LNet: Added LNI 192.168.202.107@tcp105 [8/256/0/180] [11646.279202] LNet: Added LNI 192.168.202.107@tcp106 [8/256/0/180] [11647.604048] LNet: Added LNI 192.168.202.107@tcp107 [8/256/0/180] [11648.459467] LNet: Added LNI 192.168.202.107@tcp108 [8/256/0/180] [11649.403966] LNet: Added LNI 192.168.202.107@tcp109 [8/256/0/180] [11650.392268] LNet: Added LNI 192.168.202.107@tcp110 [8/256/0/180] [11651.371033] LNet: Added LNI 192.168.202.107@tcp111 [8/256/0/180] [11652.284889] LNet: Added LNI 192.168.202.107@tcp112 [8/256/0/180] [11653.178987] LNet: Added LNI 192.168.202.107@tcp113 [8/256/0/180] [11654.230288] LNet: Added LNI 192.168.202.107@tcp114 [8/256/0/180] [11655.245644] LNet: Added LNI 192.168.202.107@tcp115 [8/256/0/180] [11656.373946] LNet: Added LNI 192.168.202.107@tcp116 [8/256/0/180] [11657.217235] LNet: Added LNI 192.168.202.107@tcp117 [8/256/0/180] [11658.103962] LNet: Added LNI 192.168.202.107@tcp118 [8/256/0/180] [11659.178249] LNet: Added LNI 192.168.202.107@tcp119 [8/256/0/180] [11660.332815] LNet: Added LNI 192.168.202.107@tcp120 [8/256/0/180] [11661.415467] LNet: Added LNI 192.168.202.107@tcp121 [8/256/0/180] [11662.472299] LNet: Added LNI 192.168.202.107@tcp122 [8/256/0/180] [11663.340797] LNet: Added LNI 192.168.202.107@tcp123 [8/256/0/180] [11664.418604] LNet: Added LNI 192.168.202.107@tcp124 [8/256/0/180] [11665.355693] LNet: Added LNI 192.168.202.107@tcp125 [8/256/0/180] [11666.349552] LNet: Added LNI 192.168.202.107@tcp126 [8/256/0/180] [11667.188975] LNet: Added LNI 192.168.202.107@tcp127 [8/256/0/180] [11668.057107] LNet: Added LNI 192.168.202.107@tcp128 [8/256/0/180] [11668.921668] LNet: Added LNI 192.168.202.107@tcp129 [8/256/0/180] [11669.857182] LNet: Added LNI 192.168.202.107@tcp130 [8/256/0/180] [11670.683208] LNet: Added LNI 192.168.202.107@tcp131 [8/256/0/180] [11671.521882] LNet: Added LNI 192.168.202.107@tcp132 [8/256/0/180] [11672.564722] LNet: Added LNI 192.168.202.107@tcp133 [8/256/0/180] [11673.557714] LNet: Added LNI 192.168.202.107@tcp134 [8/256/0/180] [11674.358363] LNet: Added LNI 192.168.202.107@tcp135 [8/256/0/180] [11681.157386] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11681.207414] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11682.445861] Lustre: 434498:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.107@tcp: can't add NID 192.168.202.107@tcp130: rc = -75 [11682.619881] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11682.640027] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11682.755185] Lustre: lustre-MDT0000: new disk, initializing [11682.929978] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11682.957286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11687.787026] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11698.813671] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11699.021811] Lustre: 434515:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11699.141971] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11699.150257] Lustre: Skipped 1 previous similar message [11699.277753] Lustre: lustre-MDT0001: new disk, initializing [11699.312822] Lustre: 435293:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.107@tcp: can't add NID 192.168.202.107@tcp130: rc = -75 [11699.318785] Lustre: 435293:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11699.417573] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11699.455112] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11699.472488] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11703.939895] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11708.007685] Lustre: 435673:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.107@tcp: can't add NID 192.168.202.107@tcp130: rc = -75 [11708.019184] Lustre: 435673:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11711.914617] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11716.051197] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11723.343750] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11723.755895] Lustre: lustre-OST0000: new disk, initializing [11723.771824] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11723.781646] Lustre: 436258:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11723.883744] Lustre: 436263:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.107@tcp: can't add NID 192.168.202.107@tcp130: rc = -75 [11723.892391] Lustre: 436263:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [11723.949510] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11729.484635] Lustre: 436532:0:(lustre_peer.c:105:class_add_uuid()) 192.168.202.107@tcp: can't add NID 192.168.202.107@tcp130: rc = -75 [11729.493284] Lustre: 436532:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [11729.554268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11729.573054] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11729.677824] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11730.863057] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11738.627544] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11750.369906] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11750.373619] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11750.375959] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11750.380902] Lustre: Skipped 1 previous similar message [11750.398696] Lustre: Skipped 1 previous similar message [11752.750584] Lustre: server umount lustre-OST0000 complete [11760.610158] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11760.630926] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11763.437922] Lustre: server umount lustre-MDT0000 complete [11767.167741] LustreError: 434506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780347096 with bad export cookie 10379249704445845799 [11767.181737] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11767.499069] Lustre: server umount lustre-MDT0001 complete [11779.970178] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [11782.920555] Key type lgssc unregistered [11783.366826] LNet: 438042:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11783.376265] LNetError: 438042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11783.386838] LNet: Removed LNI 192.168.202.107@tcp [11784.416145] LNet: Removed LNI 192.168.202.107@tcp100 [11784.419322] LNetError: 438042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11784.428529] LNetError: 438042:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [11784.437485] LNet: Removed LNI 192.168.202.107@tcp101 [11784.443444] LNet: Removed LNI 192.168.202.107@tcp102 [11784.447607] LNet: Removed LNI 192.168.202.107@tcp103 [11784.451337] LNet: Removed LNI 192.168.202.107@tcp104 [11784.455273] LNet: Removed LNI 192.168.202.107@tcp105 [11784.464249] LNet: Removed LNI 192.168.202.107@tcp106 [11784.468525] LNet: Removed LNI 192.168.202.107@tcp107 [11784.471309] LNet: Removed LNI 192.168.202.107@tcp108 [11784.474242] LNet: Removed LNI 192.168.202.107@tcp109 [11784.477398] LNet: Removed LNI 192.168.202.107@tcp110 [11784.482476] LNet: Removed LNI 192.168.202.107@tcp111 [11784.489338] LNet: Removed LNI 192.168.202.107@tcp112 [11784.498639] LNet: Removed LNI 192.168.202.107@tcp113 [11784.507199] LNet: Removed LNI 192.168.202.107@tcp114 [11784.512487] LNet: Removed LNI 192.168.202.107@tcp115 [11784.518054] LNet: Removed LNI 192.168.202.107@tcp116 [11784.521628] LNet: Removed LNI 192.168.202.107@tcp117 [11784.527764] LNet: Removed LNI 192.168.202.107@tcp118 [11784.530316] LNet: Removed LNI 192.168.202.107@tcp119 [11784.533967] LNet: Removed LNI 192.168.202.107@tcp120 [11784.538284] LNet: Removed LNI 192.168.202.107@tcp121 [11784.542075] LNet: Removed LNI 192.168.202.107@tcp122 [11784.545407] LNet: Removed LNI 192.168.202.107@tcp123 [11784.551622] LNet: Removed LNI 192.168.202.107@tcp124 [11784.555118] LNet: Removed LNI 192.168.202.107@tcp125 [11784.562210] LNet: Removed LNI 192.168.202.107@tcp126 [11784.568822] LNet: Removed LNI 192.168.202.107@tcp127 [11784.573772] LNet: Removed LNI 192.168.202.107@tcp128 [11784.577273] LNet: Removed LNI 192.168.202.107@tcp129 [11784.580368] LNet: Removed LNI 192.168.202.107@tcp130 [11784.586494] LNet: Removed LNI 192.168.202.107@tcp131 [11784.588382] LNet: Removed LNI 192.168.202.107@tcp132 [11784.592815] LNet: Removed LNI 192.168.202.107@tcp133 [11784.595679] LNet: Removed LNI 192.168.202.107@tcp134 [11784.607586] LNet: Removed LNI 192.168.202.107@tcp135 [11785.481475] Key type .llcrypt unregistered [11785.483788] Key type ._llcrypt unregistered [11808.811791] Key type ._llcrypt registered [11808.815657] Key type .llcrypt registered [11808.964521] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [11825.554791] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [11827.078375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11827.091043] alg: No test for adler32 (adler32-zlib) [11828.231905] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [11828.450646] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [11830.113469] Key type lgssc registered [11831.241783] Lustre: Echo OBD driver; http://www.lustre.org/ [11870.101101] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 16:53:17 (1780347197) [11889.188536] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [11892.357614] Key type lgssc unregistered [11892.596636] LNet: 442076:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11892.610303] LNetError: 442076:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [11892.626377] LNet: Removed LNI 192.168.202.107@tcp [11893.367173] Key type .llcrypt unregistered [11893.369054] Key type ._llcrypt unregistered [11910.673715] Key type ._llcrypt registered [11910.679951] Key type .llcrypt registered [11910.823540] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [11911.915438] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11912.161211] alg: No test for adler32 (adler32-zlib) [11913.281297] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [11913.508879] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [11915.184169] Key type lgssc registered [11916.174797] Lustre: Echo OBD driver; http://www.lustre.org/ [11925.303838] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11925.331635] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11926.669956] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11926.695601] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11926.744877] Lustre: lustre-MDT0000: new disk, initializing [11926.831559] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11926.842907] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11930.848900] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11941.706845] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11941.821468] Lustre: 443270:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11941.883709] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11941.889599] Lustre: Skipped 1 previous similar message [11942.001863] Lustre: lustre-MDT0001: new disk, initializing [11942.108921] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11942.149196] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11942.162169] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11947.117951] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11955.014547] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [11959.021933] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [11965.994617] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11966.367868] Lustre: lustre-OST0000: new disk, initializing [11966.379685] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11966.394683] Lustre: 445018:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11966.524868] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11973.075330] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [11974.174484] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11974.183365] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11974.280990] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11980.622633] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [11984.351701] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing lnet_if_list [11989.284656] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [11989.318939] LNet: Added LNI 192.168.202.107@tcp10 [8/256/0/180] [11994.980225] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [11994.999943] LNet: Added LNI 192.168.202.107@tcp11 [8/256/0/180] [11999.202851] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [11999.235689] LNet: Added LNI 192.168.202.107@tcp12 [8/256/0/180] [12003.549778] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12003.571211] LNet: Added LNI 192.168.202.107@tcp13 [8/256/0/180] [12007.744230] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12007.802620] LNet: Added LNI 192.168.202.107@tcp14 [8/256/0/180] [12011.635726] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12011.657922] LNet: Added LNI 192.168.202.107@tcp15 [8/256/0/180] [12015.162626] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12015.197751] LNet: Added LNI 192.168.202.107@tcp16 [8/256/0/180] [12019.089241] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12019.108899] LNet: Added LNI 192.168.202.107@tcp17 [8/256/0/180] [12023.028632] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12023.064312] LNet: Added LNI 192.168.202.107@tcp18 [8/256/0/180] [12027.852969] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12027.872439] LNet: Added LNI 192.168.202.107@tcp19 [8/256/0/180] [12032.564562] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12032.599310] LNet: Added LNI 192.168.202.107@tcp20 [8/256/0/180] [12038.029671] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12038.059725] LNet: Added LNI 192.168.202.107@tcp21 [8/256/0/180] [12042.883866] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12042.930289] LNet: Added LNI 192.168.202.107@tcp22 [8/256/0/180] [12049.612598] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12049.635869] LNet: Added LNI 192.168.202.107@tcp23 [8/256/0/180] [12055.725973] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12055.750463] LNet: Added LNI 192.168.202.107@tcp24 [8/256/0/180] [12061.364062] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12061.413391] LNet: Added LNI 192.168.202.107@tcp25 [8/256/0/180] [12066.575303] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12066.612940] LNet: Added LNI 192.168.202.107@tcp26 [8/256/0/180] [12072.225585] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12072.252531] LNet: Added LNI 192.168.202.107@tcp27 [8/256/0/180] [12078.230931] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12078.256569] LNet: Added LNI 192.168.202.107@tcp28 [8/256/0/180] [12084.377315] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12084.398206] LNet: Added LNI 192.168.202.107@tcp29 [8/256/0/180] [12090.962334] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [12091.006715] LNet: Added LNI 192.168.202.107@tcp30 [8/256/0/180] [12098.006608] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [12098.024603] LNet: Added LNI 192.168.202.107@tcp31 [8/256/0/180] [12104.578315] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [12104.618751] LNet: Added LNI 192.168.202.107@tcp32 [8/256/0/180] [12111.080343] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [12111.115481] LNet: Added LNI 192.168.202.107@tcp33 [8/256/0/180] [12117.761347] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [12117.788990] LNet: Added LNI 192.168.202.107@tcp34 [8/256/0/180] [12124.333253] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [12124.357927] LNet: Added LNI 192.168.202.107@tcp35 [8/256/0/180] [12130.448225] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [12130.472718] LNet: Added LNI 192.168.202.107@tcp36 [8/256/0/180] [12136.056476] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [12136.072867] LNet: Added LNI 192.168.202.107@tcp37 [8/256/0/180] [12142.560785] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [12142.586286] LNet: Added LNI 192.168.202.107@tcp38 [8/256/0/180] [12148.382731] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [12148.404072] LNet: Added LNI 192.168.202.107@tcp39 [8/256/0/180] [12155.124719] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [12155.145905] LNet: Added LNI 192.168.202.107@tcp40 [8/256/0/180] [12161.602545] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [12161.631691] LNet: Added LNI 192.168.202.107@tcp41 [8/256/0/180] [12168.323474] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [12168.348190] LNet: Added LNI 192.168.202.107@tcp42 [8/256/0/180] [12176.230177] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [12176.253852] LNet: Added LNI 192.168.202.107@tcp43 [8/256/0/180] [12184.504502] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [12184.571809] LNet: Added LNI 192.168.202.107@tcp44 [8/256/0/180] [12192.667425] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [12192.714734] LNet: Added LNI 192.168.202.107@tcp45 [8/256/0/180] [12200.997583] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [12201.058639] LNet: Added LNI 192.168.202.107@tcp46 [8/256/0/180] [12209.041207] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [12209.074578] LNet: Added LNI 192.168.202.107@tcp47 [8/256/0/180] [12216.892749] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [12216.918436] LNet: Added LNI 192.168.202.107@tcp48 [8/256/0/180] [12225.723175] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [12225.764475] LNet: Added LNI 192.168.202.107@tcp49 [8/256/0/180] [12233.833168] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [12233.858733] LNet: Added LNI 192.168.202.107@tcp50 [8/256/0/180] [12254.700436] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12254.709800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12254.728161] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12255.718182] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12255.719636] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12255.739688] Lustre: Skipped 1 previous similar message [12255.746592] Lustre: Skipped 1 previous similar message [12257.156606] Lustre: server umount lustre-MDT0000 complete [12261.012414] LustreError: 445017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780347589 with bad export cookie 18037058380308983703 [12261.017857] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12261.030876] LustreError: 445017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12261.523506] Lustre: server umount lustre-MDT0001 complete [12279.585262] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12281.314548] Lustre: 442543:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780347594/real 1780347594] req@000000008a307b1d x1866829389796224/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780347610 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12281.343085] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12282.401207] Lustre: 442543:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780347595/real 1780347595] req@000000006c983560 x1866829389796352/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780347611 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12286.499270] Lustre: 442546:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780347599/real 1780347599] req@00000000cf7f5088 x1866829389796480/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780347615 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12287.407330] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12292.704148] Lustre: 442546:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780347605/real 1780347605] req@000000009ee2bd3f x1866829389796736/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780347621 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12297.698083] Lustre: 442544:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1780347610/real 1780347610] req@00000000d9c93ac3 x1866829389796864/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1780347626 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [12307.041638] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xfa5081084f4cb9b5 [12307.063837] Lustre: MGC192.168.202.107@tcp: Connection restored to 0@lo (at 0@lo) [12307.569414] LustreError: 445010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12307.603394] LustreError: 445010:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [12307.639186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12310.760271] LustreError: 445011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12310.770941] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12311.635707] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12312.803633] LustreError: 445010:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12318.185368] LustreError: 445011:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12319.974808] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12320.268592] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12324.293757] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12327.406866] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [12329.824882] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12334.084516] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12352.995030] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12353.030290] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12353.038930] Lustre: Skipped 2 previous similar messages [12358.114976] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12358.119963] Lustre: Skipped 1 previous similar message [12358.417280] Lustre: server umount lustre-OST0000 complete [12363.233409] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12363.236850] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12363.250802] Lustre: Skipped 2 previous similar messages [12363.262845] Lustre: Skipped 1 previous similar message [12368.270817] Lustre: server umount lustre-MDT0000 complete [12368.356348] LustreError: 468416:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12368.368869] LustreError: 468416:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [12372.120655] LustreError: 443263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780347701 with bad export cookie 18037058380308986293 [12372.123452] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12372.129395] LustreError: 443263:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12372.494259] Lustre: server umount lustre-MDT0001 complete [12384.560851] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [12387.311643] Key type lgssc unregistered [12387.717445] LNet: 470984:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12387.733745] LNetError: 470984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12388.770612] LNet: Removed LNI 192.168.202.107@tcp [12388.773245] LNetError: 470984:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12388.783154] LNet: Removed LNI 192.168.202.107@tcp10 [12388.786554] LNet: Removed LNI 192.168.202.107@tcp11 [12388.789656] LNet: Removed LNI 192.168.202.107@tcp12 [12388.792434] LNet: Removed LNI 192.168.202.107@tcp13 [12388.795269] LNet: Removed LNI 192.168.202.107@tcp14 [12388.797507] LNet: Removed LNI 192.168.202.107@tcp15 [12388.800271] LNet: Removed LNI 192.168.202.107@tcp16 [12388.803636] LNet: Removed LNI 192.168.202.107@tcp17 [12388.805175] LNet: Removed LNI 192.168.202.107@tcp18 [12388.806274] LNet: Removed LNI 192.168.202.107@tcp19 [12388.810527] LNet: Removed LNI 192.168.202.107@tcp20 [12388.812330] LNet: Removed LNI 192.168.202.107@tcp21 [12388.814839] LNet: Removed LNI 192.168.202.107@tcp22 [12388.815898] LNet: Removed LNI 192.168.202.107@tcp23 [12388.816794] LNet: Removed LNI 192.168.202.107@tcp24 [12388.817695] LNet: Removed LNI 192.168.202.107@tcp25 [12388.822136] LNet: Removed LNI 192.168.202.107@tcp26 [12388.824666] LNet: Removed LNI 192.168.202.107@tcp27 [12388.825888] LNet: Removed LNI 192.168.202.107@tcp28 [12388.826758] LNet: Removed LNI 192.168.202.107@tcp29 [12388.828037] LNet: Removed LNI 192.168.202.107@tcp30 [12388.831524] LNet: Removed LNI 192.168.202.107@tcp31 [12388.835598] LNet: Removed LNI 192.168.202.107@tcp32 [12388.837506] LNet: Removed LNI 192.168.202.107@tcp33 [12388.844196] LNet: Removed LNI 192.168.202.107@tcp34 [12388.848636] LNet: Removed LNI 192.168.202.107@tcp35 [12388.852217] LNet: Removed LNI 192.168.202.107@tcp36 [12388.854655] LNet: Removed LNI 192.168.202.107@tcp37 [12388.857440] LNet: Removed LNI 192.168.202.107@tcp38 [12388.859432] LNet: Removed LNI 192.168.202.107@tcp39 [12388.861500] LNet: Removed LNI 192.168.202.107@tcp40 [12388.863937] LNet: Removed LNI 192.168.202.107@tcp41 [12388.866939] LNet: Removed LNI 192.168.202.107@tcp42 [12388.869929] LNet: Removed LNI 192.168.202.107@tcp43 [12388.871351] LNet: Removed LNI 192.168.202.107@tcp44 [12388.872921] LNet: Removed LNI 192.168.202.107@tcp45 [12388.874411] LNet: Removed LNI 192.168.202.107@tcp46 [12388.877684] LNet: Removed LNI 192.168.202.107@tcp47 [12388.880447] LNet: Removed LNI 192.168.202.107@tcp48 [12388.881734] LNet: Removed LNI 192.168.202.107@tcp49 [12388.890569] LNet: Removed LNI 192.168.202.107@tcp50 [12389.539294] Key type .llcrypt unregistered [12389.541572] Key type ._llcrypt unregistered [12408.640496] Key type ._llcrypt registered [12408.642440] Key type .llcrypt registered [12408.747771] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [12423.174243] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12424.459973] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12424.528596] alg: No test for adler32 (adler32-zlib) [12425.585067] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [12425.767037] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [12427.400165] Key type lgssc registered [12428.516303] Lustre: Echo OBD driver; http://www.lustre.org/ [12458.704941] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12458.731481] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12459.959697] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12459.992923] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12460.059583] Lustre: lustre-MDT0000: new disk, initializing [12460.129610] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12460.142767] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12463.133716] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12472.139687] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12472.314888] Lustre: 474162:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12472.438060] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12472.444752] Lustre: Skipped 1 previous similar message [12472.564818] Lustre: lustre-MDT0001: new disk, initializing [12472.638450] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12472.675670] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12472.688080] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12476.753862] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12483.814142] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12486.908569] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12492.347024] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12492.499242] Lustre: *** cfs_fail_loc=90f, val=0*** [12492.565571] Lustre: lustre-OST0000: new disk, initializing [12492.567480] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12492.570665] Lustre: 475952:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12492.639550] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12497.758633] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12500.016497] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12500.027715] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12500.107113] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12503.859106] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12510.697258] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12510.698158] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12510.724427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12515.297348] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12515.306053] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12515.322264] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12516.737194] Lustre: server umount lustre-OST0000 complete [12520.931948] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12520.941225] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12520.941235] Lustre: Skipped 1 previous similar message [12520.949904] Lustre: Skipped 1 previous similar message [12526.055334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12526.071691] Lustre: Skipped 2 previous similar messages [12526.372968] Lustre: server umount lustre-MDT0000 complete [12528.871138] LustreError: 474154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780347857 with bad export cookie 675718929687142500 [12528.881629] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12528.884375] LustreError: 474154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12529.100354] Lustre: server umount lustre-MDT0001 complete [12536.711975] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [12538.790935] Key type lgssc unregistered [12539.057567] LNet: 477723:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12539.065977] LNetError: 477723:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12539.085309] LNet: Removed LNI 192.168.202.107@tcp [12539.823268] Key type .llcrypt unregistered [12539.826356] Key type ._llcrypt unregistered [12554.420160] Key type ._llcrypt registered [12554.421809] Key type .llcrypt registered [12554.489901] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12555.158585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12555.224025] alg: No test for adler32 (adler32-zlib) [12556.211485] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [12556.390551] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [12558.034405] Key type lgssc registered [12558.827709] Lustre: Echo OBD driver; http://www.lustre.org/ [12565.114882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12565.135637] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12566.493634] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12569.756883] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12576.495844] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12576.811690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12580.119494] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12584.812978] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12588.711639] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12594.816720] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12595.046255] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12601.589164] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12606.978369] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12619.778198] loop0: detected capacity change from 0 to 204800000 [12619.983548] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12619.995097] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12620.007532] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12624.971255] loop0: detected capacity change from 0 to 204800000 [12625.126764] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12625.135842] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12625.142398] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [12629.237699] loop0: detected capacity change from 0 to 204800000 [12633.883039] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12633.978665] Lustre: fs15246-OST0000: new disk, initializing [12633.982600] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [12633.988138] Lustre: 482210:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12634.027280] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12639.426129] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12648.082715] loop1: detected capacity change from 0 to 204800000 [12652.429703] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12652.499504] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [12652.547675] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [12652.620977] Lustre: fs15246-MDT0000: new disk, initializing [12652.734277] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12652.750390] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12652.827587] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [12653.858354] Lustre: 483314:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@000000001f260281 x1866830063905152/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1780347998 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [12656.178850] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12662.949863] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [12662.956853] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [12662.984142] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [12675.576330] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12675.585020] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [12677.541890] Lustre: server umount fs15246-MDT0000 complete [12680.685894] LustreError: 480319:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12680.712160] LustreError: 480319:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [12684.864804] Lustre: server umount fs15246-OST0000 complete [12696.037081] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12696.048309] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12696.052617] Lustre: Skipped 1 previous similar message [12699.829212] Lustre: server umount lustre-OST0000 complete [12704.738529] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12704.743481] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12704.747776] Lustre: Skipped 1 previous similar message [12704.752241] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12708.524546] Lustre: server umount lustre-MDT0000 complete [12711.361210] LustreError: 478937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780348040 with bad export cookie 16110051027598289801 [12711.363463] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12711.367526] LustreError: 478937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12711.567095] Lustre: server umount lustre-MDT0001 complete [12719.109218] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [12720.765582] Key type lgssc unregistered [12720.976573] LNet: 486375:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12720.981387] LNetError: 486375:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12720.996201] LNet: Removed LNI 192.168.202.107@tcp [12721.481946] Key type .llcrypt unregistered [12721.483582] Key type ._llcrypt unregistered [12734.416162] Key type ._llcrypt registered [12734.418944] Key type .llcrypt registered [12734.521899] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12735.531601] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12735.561932] alg: No test for adler32 (adler32-zlib) [12736.578494] Lustre: Lustre: Build Version: 2.17.53_34_g0f61aab [12736.746463] LNet: Added LNI 192.168.202.107@tcp [8/256/0/180] [12738.448174] Key type lgssc registered [12739.245935] Lustre: Echo OBD driver; http://www.lustre.org/ [12767.482795] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [12775.179575] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12799.707850] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12799.738991] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12800.977777] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12801.002749] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12801.069556] Lustre: lustre-MDT0000: new disk, initializing [12801.133827] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12801.156519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12804.698979] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12812.928703] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12813.054502] Lustre: 490765:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12813.147267] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12813.154772] Lustre: Skipped 1 previous similar message [12813.264572] Lustre: lustre-MDT0001: new disk, initializing [12813.399372] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12813.428838] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12813.450844] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12816.956648] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12822.549446] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [12825.277469] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [12830.035130] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12830.223090] Lustre: lustre-OST0000: new disk, initializing [12830.225235] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12830.231043] Lustre: 492507:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12830.305259] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12834.977715] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_default_debug -1 all [12840.010364] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12840.016325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12840.074078] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12841.050517] Lustre: DEBUG MARKER: oleg207-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [12844.093853] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [12844.210982] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [12847.518706] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [12847.680793] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [12850.663414] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [12850.666423] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12850.671971] LustreError: Skipped 1 previous similar message [12850.672822] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12850.697211] Lustre: Skipped 1 previous similar message [12855.456355] Lustre: server umount lustre-OST0000 complete [12859.363486] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [12859.367227] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12859.374810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12859.377400] Lustre: Skipped 1 previous similar message [12860.904982] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12860.909714] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12864.252476] Lustre: server umount lustre-MDT0000 complete [12867.142996] LustreError: 490757:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1780348196 with bad export cookie 10868348136440120951 [12867.153746] LustreError: MGC192.168.202.107@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12867.396387] Lustre: server umount lustre-MDT0001 complete [12885.970725] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing set_hostid [12893.241243] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing load_modules_local [12917.055932] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 12643 sec ========= 17:10:45 (1780348245) [12918.503361] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 17:10:46 (1780348246) === [12921.091969] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 17:10:49 (1780348249) === [12948.987454] Lustre: DEBUG MARKER: oleg207-server.virtnet: executing unload_modules_local [12950.739836] Key type lgssc unregistered [12950.959740] LNet: 498945:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12950.968758] LNetError: 498945:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [12950.982994] LNet: Removed LNI 192.168.202.107@tcp [12951.494935] Key type .llcrypt unregistered [12951.496898] Key type ._llcrypt unregistered