[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 450369189 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2829652K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524580K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001011] APIC: Switch to symmetric I/O mode setup [ 0.003142] x2apic enabled [ 0.004007] Switched APIC routing to physical x2apic. [ 0.005014] kvm-guest: setup PV IPIs [ 0.008534] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.009000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.009020] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.010011] pid_max: default: 32768 minimum: 301 [ 0.011117] LSM: Security Framework initializing [ 0.012054] Yama: becoming mindful. [ 0.013030] SELinux: Initializing. [ 0.014071] *** VALIDATE selinux *** [ 0.022140] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.026741] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.027158] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028251] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.030143] *** VALIDATE tmpfs *** [ 0.031555] *** VALIDATE proc *** [ 0.032292] *** VALIDATE cgroup *** [ 0.034009] *** VALIDATE cgroup2 *** [ 0.035264] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.037073] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.038011] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.039047] Spectre V2 : User space: Vulnerable [ 0.040011] Speculative Store Bypass: Vulnerable [ 0.043211] debug: unmapping init [mem 0xffffffff84859000-0xffffffff84860fff] [ 0.045523] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.047299] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.048030] ... version: 2 [ 0.049013] ... bit width: 48 [ 0.050012] ... generic registers: 4 [ 0.051014] ... value mask: 0000ffffffffffff [ 0.052017] ... max period: 00007fffffffffff [ 0.053016] ... fixed-purpose events: 3 [ 0.054013] ... event mask: 000000070000000f [ 0.055357] rcu: Hierarchical SRCU implementation. [ 0.057672] smp: Bringing up secondary CPUs ... [ 0.058653] x86: Booting SMP configuration: [ 0.059025] .... node #0, CPUs: #1 #2 #3 [ 0.063532] smp: Brought up 1 node, 4 CPUs [ 0.065011] smpboot: Max logical packages: 1 [ 0.066017] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.227913] node 0 deferred pages initialised in 159ms [ 0.230562] devtmpfs: initialized [ 0.232315] x86/mm: Memory block size: 128MB [ 0.235572] gcov: version magic: 0x41383552 [ 0.238334] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.239096] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.241339] pinctrl core: initialized pinctrl subsystem [ 0.243185] [ 0.243850] ************************************************************* [ 0.246010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.249013] ** ** [ 0.251009] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.253009] ** ** [ 0.255008] ** This means that this kernel is built to expose internal ** [ 0.257009] ** IOMMU data structures, which may compromise security on ** [ 0.258009] ** your system. ** [ 0.259000] ** ** [ 0.260008] ** If you see this message and you are not debugging the ** [ 0.263013] ** kernel, report this immediately to your vendor! ** [ 0.264009] ** ** [ 0.266009] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.267009] ************************************************************* [ 0.269616] NET: Registered protocol family 16 [ 0.271633] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.275084] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.277072] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.280205] cpuidle: using governor menu [ 0.281783] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.284639] PCI: Using configuration type 1 for base access [ 0.286139] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.295058] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.296021] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.298083] cryptd: max_cpu_qlen set to 1000 [ 0.302030] ACPI: Added _OSI(Module Device) [ 0.303010] ACPI: Added _OSI(Processor Device) [ 0.304011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.306011] ACPI: Added _OSI(Processor Aggregator Device) [ 0.309586] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.315386] ACPI: Interpreter enabled [ 0.317075] ACPI: PM: (supports S0 S3 S4 S5) [ 0.318011] ACPI: Using IOAPIC for interrupt routing [ 0.320099] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.323377] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.332389] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.334043] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.337018] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.340072] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.345302] acpiphp: Slot [2] registered [ 0.347084] acpiphp: Slot [5] registered [ 0.347979] acpiphp: Slot [6] registered [ 0.349102] acpiphp: Slot [7] registered [ 0.350075] acpiphp: Slot [8] registered [ 0.352099] acpiphp: Slot [9] registered [ 0.353109] acpiphp: Slot [10] registered [ 0.354091] acpiphp: Slot [3] registered [ 0.355082] acpiphp: Slot [4] registered [ 0.357105] acpiphp: Slot [11] registered [ 0.358082] acpiphp: Slot [12] registered [ 0.358940] acpiphp: Slot [13] registered [ 0.360101] acpiphp: Slot [14] registered [ 0.362106] acpiphp: Slot [15] registered [ 0.363093] acpiphp: Slot [16] registered [ 0.365085] acpiphp: Slot [17] registered [ 0.366103] acpiphp: Slot [18] registered [ 0.368125] acpiphp: Slot [19] registered [ 0.369098] acpiphp: Slot [20] registered [ 0.371099] acpiphp: Slot [21] registered [ 0.372084] acpiphp: Slot [22] registered [ 0.373083] acpiphp: Slot [23] registered [ 0.375081] acpiphp: Slot [24] registered [ 0.376086] acpiphp: Slot [25] registered [ 0.377113] acpiphp: Slot [26] registered [ 0.379091] acpiphp: Slot [27] registered [ 0.380063] acpiphp: Slot [28] registered [ 0.381101] acpiphp: Slot [29] registered [ 0.382079] acpiphp: Slot [30] registered [ 0.384076] acpiphp: Slot [31] registered [ 0.385000] PCI host bridge to bus 0000:00 [ 0.385000] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.387029] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.389019] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.391031] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.393030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.396030] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.398157] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.399875] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.403815] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.413727] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.419035] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.422026] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.423020] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.426024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.429694] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.433097] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.435051] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.437757] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.442016] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.455945] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.459014] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.462707] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.474018] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.479017] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.493021] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.503294] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.509021] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.515016] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.533017] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.544189] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.551021] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.556012] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.570015] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.578743] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.584017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.592022] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.606019] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.618178] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.624015] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.634014] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.650015] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.662581] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.669018] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.676017] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.691017] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.702007] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.704319] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.705260] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.707297] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.709160] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.713468] iommu: Default domain type: Passthrough [ 0.715506] SCSI subsystem initialized [ 0.717134] ACPI: bus type USB registered [ 0.719139] usbcore: registered new interface driver usbfs [ 0.721073] usbcore: registered new interface driver hub [ 0.723083] usbcore: registered new device driver usb [ 0.725139] pps_core: LinuxPPS API ver. 1 registered [ 0.727011] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.730047] PTP clock support registered [ 0.732075] EDAC MC: Ver: 3.0.0 [ 0.733453] PCI: Using ACPI for IRQ routing [ 0.735804] NetLabel: Initializing [ 0.737010] NetLabel: domain hash size = 128 [ 0.738010] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.739072] NetLabel: unlabeled traffic allowed by default [ 0.742098] vgaarb: loaded [ 0.743182] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.745014] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.750015] clocksource: Switched to clocksource kvm-clock [ 0.848478] VFS: Disk quotas dquot_6.6.0 [ 0.850062] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.852979] *** VALIDATE ramfs *** [ 0.854125] *** VALIDATE hugetlbfs *** [ 0.855555] pnp: PnP ACPI init [ 0.858033] pnp: PnP ACPI: found 6 devices [ 0.874338] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.877828] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.880217] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.882655] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.885346] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.887916] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.891044] NET: Registered protocol family 2 [ 0.893489] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.897713] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.901608] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.906729] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.910200] TCP: Hash tables configured (established 65536 bind 65536) [ 0.912457] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.915142] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.918123] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.920874] NET: Registered protocol family 1 [ 0.923603] RPC: Registered named UNIX socket transport module. [ 0.925811] RPC: Registered udp transport module. [ 0.927432] RPC: Registered tcp transport module. [ 0.928577] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.931129] NET: Registered protocol family 44 [ 0.932944] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.935275] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.937836] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.939856] PCI: CLS 0 bytes, default 64 [ 0.941461] Unpacking initramfs... [ 2.282398] debug: unmapping init [mem 0xffff9ae6bcc54000-0xffff9ae6bffbffff] [ 2.285743] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.288341] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.291233] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.760948] Initialise system trusted keyrings [ 2.762159] Key type blacklist registered [ 2.764665] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.772825] zbud: loaded [ 2.775897] *** VALIDATE nfs *** [ 2.777120] *** VALIDATE nfs4 *** [ 2.778935] pstore: using deflate compression [ 2.782374] Platform Keyring initialized [ 2.882804] NET: Registered protocol family 38 [ 2.884757] Key type asymmetric registered [ 2.886025] Asymmetric key parser 'x509' registered [ 2.887924] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.890753] io scheduler mq-deadline registered [ 2.892612] io scheduler kyber registered [ 2.894558] io scheduler bfq registered [ 2.896178] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.899472] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.902264] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.904291] ACPI: Power Button [PWRF] [ 2.908485] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.914547] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.927760] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 2.935186] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 2.956659] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 2.983572] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.013255] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.017944] Non-volatile memory driver v1.3 [ 3.019474] Linux agpgart interface v0.103 [ 3.053873] virtio_blk virtio1: [vda] 136552 512-byte logical blocks (69.9 MB/66.7 MiB) [ 3.057526] vda: detected capacity change from 0 to 69914624 [ 3.073351] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.076242] vdb: detected capacity change from 0 to 1073741824 [ 3.091511] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.094877] vdc: detected capacity change from 0 to 2621440000 [ 3.113426] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.116653] vdd: detected capacity change from 0 to 2621440000 [ 3.131526] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.134541] vde: detected capacity change from 0 to 4294967296 [ 3.148475] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.151401] vdf: detected capacity change from 0 to 4294967296 [ 3.160373] libphy: Fixed MDIO Bus: probed [ 3.164566] usbcore: registered new interface driver usbserial_generic [ 3.166746] usbserial: USB Serial support registered for generic [ 3.169581] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.174130] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.175763] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.178555] mousedev: PS/2 mouse device common for all mice [ 3.180569] rtc_cmos 00:05: RTC can wake from S4 [ 3.183194] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.186962] rtc_cmos 00:05: registered as rtc0 [ 3.188177] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.189693] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.194459] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.196173] intel_pstate: CPU model not supported [ 3.202494] hid: raw HID events driver (C) Jiri Kosina [ 3.204967] usbcore: registered new interface driver usbhid [ 3.207074] usbhid: USB HID core driver [ 3.208831] drop_monitor: Initializing network drop monitor service [ 3.211202] Initializing XFRM netlink socket [ 3.213555] NET: Registered protocol family 10 [ 3.216961] Segment Routing with IPv6 [ 3.218454] NET: Registered protocol family 17 [ 3.220118] mpls_gso: MPLS GSO support [ 3.226400] RAS: Correctable Errors collector initialized. [ 3.228762] AVX version of gcm_enc/dec engaged. [ 3.230335] AES CTR mode by8 optimization enabled [ 3.313775] sched_clock: Marking stable (3313685161, 0)->(4144405917, -830720756) [ 3.316392] registered taskstats version 1 [ 3.317530] Loading compiled-in X.509 certificates [ 3.320522] zswap: loaded using pool lzo/zbud [ 3.343617] Key type big_key registered [ 3.353745] Key type encrypted registered [ 3.354926] ima: No TPM chip found, activating TPM-bypass! [ 3.356887] ima: Allocated hash algorithm: sha1 [ 3.358058] ima: No architecture policies found [ 3.359869] evm: Initialising EVM extended attributes: [ 3.361606] evm: security.selinux [ 3.362789] evm: security.ima [ 3.363868] evm: security.capability [ 3.365165] evm: HMAC attrs: 0x1 [ 3.367455] rtc_cmos 00:05: setting system clock to 2026-06-11 22:55:21 UTC (1781218521) [ 3.374514] debug: unmapping init [mem 0xffffffff85803000-0xffffffff859fffff] [ 3.377434] debug: unmapping init [mem 0xffffffff84582000-0xffffffff84858fff] [ 3.385856] Write protecting the kernel read-only data: 28672k [ 3.389931] debug: unmapping init [mem 0xffffffff82c03000-0xffffffff82dfffff] [ 3.394127] debug: unmapping init [mem 0xffffffff83514000-0xffffffff835fffff] [ 3.437140] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.445690] systemd[1]: Detected virtualization kvm. [ 3.448051] systemd[1]: Detected architecture x86-64. [ 3.450178] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.479166] systemd[1]: No hostname configured. [ 3.480969] systemd[1]: Set hostname to . [ 3.483338] random: systemd: uninitialized urandom read (16 bytes read) [ 3.485984] systemd[1]: Initializing machine ID from random generator. [ 3.537111] random: ln: uninitialized urandom read (6 bytes read) [ 3.620983] random: systemd: uninitialized urandom read (16 bytes read) [ 3.623846] systemd[1]: Listening on Journal Socket. [ OK ] Listening on Journal Socket. [ 3.632560] systemd[1]: Starting Create list of required static device nodes for the current kernel... Starting Create list of required st…ce nodes for the current kernel... [ 3.643516] systemd[1]: Starting Setup Virtual Console... Starting Setup Virtual Console... [ OK ] Listening on Journal Socket (/dev/log). Starting Journal Service... [ OK ] Reached target Slices. [ OK ] Started Memstrack Anylazing Service. [ OK ] Reached target Timers. [ OK ] Listening on udev Kernel Socket. Starting Apply Kernel Variables... [ OK ] Reached target Local File Systems. Starting Create Volatile Files and Directories... [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Reached target Paths. [ OK ] Reached target Initrd Root Device. [ OK ] Listening on udev Control Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Swap. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create Volatile Files and Directories. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Journal Service. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.215028] device-mapper: uevent: version 1.0.3 [ 4.217219] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. Starting dracut initqueue hook... [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ 4.848375] virtio_net virtio0 ens2: renamed from eth0 [ 4.858176] scsi host0: ata_piix [ 4.879007] random: fast init done [ 4.927409] scsi host1: ata_piix [ 4.929425] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 4.933176] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 8.462641] dracut-initqueue[581]: RTNETLINK answers: File exists [ 9.870766] random: crng init done [ 9.872160] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Mounting /sysroot... [ 10.239638] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Initrd Default Target. Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Timers. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Slices. [ OK ] Stopped target Paths. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. Stopping udev Kernel Device Manager... [ OK ] Stopped target Swap. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. [ OK ] Stopped Apply Kernel Variables. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.440164] printk: systemd: 25 output lines suppressed due to ratelimiting [ 11.731924] SELinux: Disabled at runtime. [ 11.794119] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.803055] systemd[1]: Detected virtualization kvm. [ 11.804957] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 12.341956] systemd[1]: initrd-switch-root.service: Succeeded. [ 12.345030] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 12.349960] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 12.355323] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 12.358639] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.370376] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.374854] systemd[1]: Listening on udev Control Socket. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Process Core Dump Socket. Mounting POSIX Message Queue File System... [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Created slice User and Session Slice. [ OK ] Listening on udev Kernel Socket. [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target rpc_pipefs.target. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Remount Root and Kernel File Systems... Starting Apply Kernel Variables... [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice system-getty.slice. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Stopped target Initrd File Systems. Activating swap /dev/disk/by-label/SWAP... [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Reached target Slices. [ OK ] Reached target Local Encrypted Volumes. Mounting Kernel Debug File System... Starting udev Coldplug all Devices... [ 12.521811] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on RPCbind Server Activation Socket. [ OK ] Reached target RPC Port Mapper. [ OK ] Created slice system-serial\x2dgetty.slice. [ OK ] Started Journal Service. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Mounted Huge Pages File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Started Apply Kernel Variables. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Mounted Kernel Debug File System. [ OK ] Reached target Swap. Starting Configure read-only root support... Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Reached target Local File Systems (Pre). Mounting /mnt... Mounting /home/green/git/lustre-release... Starting udev Kernel Device Manager... [ OK ] Mounted /mnt. [ 12.804537] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Coldplug all Devices. [ OK ] Started udev Kernel Device Manager. [ 13.143106] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 13.151894] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 13.340230] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 13.353398] EDAC sbridge: Ver: 1.1.2 [ 15.114287] Key type dns_resolver registered [ 15.462179] NFS: Registering the id_resolver key type [ 15.467696] Key type id_resolver registered [ 15.472198] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Rebuild Dynamic Linker Cache... Starting Mark the need to relabel after reboot... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Started dnf makecache --timer. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Reached target sshd-keygen.target. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Network Manager... [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting Dynamic System Tuning Daemon... Starting GSSAPI Proxy Daemon... Starting OpenSSH server daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started OpenSSH server daemon. [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started Permit User Sessions. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Notify NFS peers of a restart... Starting System Logging Service... Starting Crash recovery kernel arming... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg640-server login: [ 44.701824] libcfs: loading out-of-tree module taints kernel. [ 44.728881] Key type ._llcrypt registered [ 44.730554] Key type .llcrypt registered [ 44.791355] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 53.809394] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 54.637422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 54.648197] alg: No test for adler32 (adler32-zlib) [ 55.762821] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 56.270037] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 57.992206] Key type lgssc registered [ 59.054344] Lustre: Echo OBD driver; http://www.lustre.org/ [ 67.756296] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 86.863876] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 93.231206] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 93.251231] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 94.425826] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 94.449530] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 94.516534] Lustre: lustre-MDT0000: new disk, initializing [ 94.571986] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 94.583147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 96.609913] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 103.634765] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 103.689257] Lustre: 6479:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 103.712563] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 103.718195] Lustre: Skipped 1 previous similar message [ 103.766156] Lustre: lustre-MDT0001: new disk, initializing [ 103.802053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 103.819230] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 103.824699] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 105.779374] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 108.842168] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 113.704691] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 113.823188] Lustre: lustre-OST0000: new disk, initializing [ 113.825554] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 113.829343] Lustre: 8381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 113.859824] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 115.787919] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 115.792101] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 115.837195] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 116.811772] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 124.525320] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 124.590078] Lustre: lustre-OST0001: new disk, initializing [ 124.592363] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 124.595534] Lustre: 9436:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 124.626336] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 127.392027] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 134.134777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 134.143377] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 134.172087] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 134.406187] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 141.740803] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 144.403271] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing check_logdir /tmp/testlogs/ [ 146.519864] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing yml_node [ 148.613769] Lustre: DEBUG MARKER: Client: 2.17.53.74 [ 149.757707] Lustre: DEBUG MARKER: MDS: 2.17.53.74 [ 151.074255] Lustre: DEBUG MARKER: OSS: 2.17.53.74 [ 151.829320] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Thu Jun 11 18:57:50 EDT 2026 [ 160.037790] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 160.702347] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 175.073169] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 175.074290] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 175.081809] Lustre: Skipped 3 previous similar messages [ 175.087015] Lustre: Skipped 3 previous similar messages [ 180.192623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 180.195903] Lustre: Skipped 3 previous similar messages [ 185.312838] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 185.317681] Lustre: Skipped 3 previous similar messages [ 187.360255] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 187.461168] Lustre: server umount lustre-MDT0000 complete [ 189.107411] LustreError: 6471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218707 with bad export cookie 9707859414928571355 [ 189.111946] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 189.114416] LustreError: 6471:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 190.433920] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 190.436739] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 190.439452] Lustre: Skipped 1 previous similar message [ 190.445468] Lustre: Skipped 1 previous similar message [ 194.466113] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 194.472390] Lustre: Skipped 1 previous similar message [ 195.326645] Lustre: server umount lustre-MDT0001 complete [ 197.114270] Lustre: server umount lustre-OST0000 complete [ 198.864051] Lustre: server umount lustre-OST0001 complete [ 201.602920] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 205.431810] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 220.991333] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 221.107540] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 221.130650] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 221.176321] Lustre: lustre-MDT0000: new disk, initializing [ 221.220248] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 221.230158] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 222.908243] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 228.150337] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 228.224318] Lustre: 14943:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 228.376428] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 230.080793] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 233.780336] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 235.359154] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 238.371424] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 238.514322] Lustre: lustre-OST0000: new disk, initializing [ 238.516655] Lustre: Skipped 1 previous similar message [ 238.519307] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 238.523319] Lustre: Skipped 2 previous similar messages [ 238.526540] Lustre: 16683:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 238.571688] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 238.576527] Lustre: Skipped 1 previous similar message [ 240.132933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 240.137656] Lustre: Skipped 1 previous similar message [ 240.139628] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 240.183317] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 241.139423] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 244.885816] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 246.571150] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 246.663850] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 248.306996] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 248.402757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 250.342236] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 250.343435] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.354045] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 250.358549] Lustre: Skipped 1 previous similar message [ 255.395490] Lustre: server umount lustre-OST0000 complete [ 259.041316] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 259.045014] LustreError: Skipped 1 previous similar message [ 259.047257] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.053068] Lustre: Skipped 1 previous similar message [ 270.818542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 270.821178] Lustre: Skipped 6 previous similar messages [ 271.328471] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 271.413807] Lustre: server umount lustre-MDT0000 complete [ 273.073907] LustreError: 14933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218791 with bad export cookie 9707859414928572888 [ 273.074957] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 273.080220] LustreError: 14933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 273.244544] Lustre: server umount lustre-MDT0001 complete [ 275.044350] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 18:59:53 (1781218793) [ 276.163606] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 276.944955] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 18:59:55 (1781218795) [ 279.919301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 280.153282] LustreError: 18944:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 280.162678] LustreError: 18944:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 5 previous similar messages [ 280.188557] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 282.115473] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 285.668671] LustreError: 18945:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 286.319993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 288.546309] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 291.358241] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 293.373827] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 296.977311] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 300.203380] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 302.987120] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 315.874236] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 315.877843] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 315.882657] Lustre: Skipped 1 previous similar message [ 315.885810] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 317.484734] Lustre: server umount lustre-OST0000 complete [ 325.219229] Lustre: server umount lustre-MDT0000 complete [ 326.698511] LustreError: 18925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781218844 with bad export cookie 9707859414928573770 [ 326.701834] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 326.703379] LustreError: 18925:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 326.848646] Lustre: server umount lustre-MDT0001 complete [ 331.171061] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 332.484188] Key type lgssc unregistered [ 332.650605] LNet: 22136:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 332.660678] LNetError: 22136:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 332.672742] LNet: Removed LNI 192.168.206.140@tcp [ 333.096193] Key type .llcrypt unregistered [ 333.098362] Key type ._llcrypt unregistered [ 343.674977] Key type ._llcrypt registered [ 343.677328] Key type .llcrypt registered [ 343.733764] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 344.165919] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 344.223184] alg: No test for adler32 (adler32-zlib) [ 345.104412] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 345.210942] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 346.808206] Key type lgssc registered [ 347.292142] Lustre: Echo OBD driver; http://www.lustre.org/ [ 350.939404] loop: module loaded [ 350.965852] loop0: detected capacity change from 0 to 204800000 [ 351.034497] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 351.040227] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 351.191802] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 353.884451] loop0: detected capacity change from 0 to 204800000 [ 353.962815] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 353.967715] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 353.973849] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 356.255400] loop0: detected capacity change from 0 to 204800000 [ 358.899990] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 358.909238] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 360.047155] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 360.069283] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 360.112503] Lustre: lustre-MDT0000: new disk, initializing [ 360.149279] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 360.159405] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 361.827034] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 365.976850] loop1: detected capacity change from 0 to 204800000 [ 368.399691] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 368.521704] Lustre: lustre-OST07c6: new disk, initializing [ 368.524618] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 368.528844] Lustre: Skipped 1 previous similar message [ 368.531886] Lustre: 25279:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 368.568598] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 371.141836] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 374.803443] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 374.809476] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 374.827171] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 379.843702] Lustre: 24311:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 254 < left 264, rollback = 2 [ 379.848236] Lustre: 24311:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 379.852200] Lustre: 24311:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 379.855983] Lustre: 24311:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 379.859950] Lustre: 24311:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/4, delete: 0/0/0 [ 379.863710] Lustre: 24311:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 391.398796] Lustre: 24311:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 391.404478] Lustre: 24311:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 391.408075] Lustre: 24311:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 391.411166] Lustre: 24311:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 391.415061] Lustre: 24311:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 391.418791] Lustre: 24311:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 395.131186] Lustre: 24311:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 395.136918] Lustre: 24311:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 395.140430] Lustre: 24311:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 395.143507] Lustre: 24311:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 395.148219] Lustre: 24311:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 395.151670] Lustre: 24311:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 395.155083] Lustre: 24311:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 395.159522] Lustre: 24311:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 395.162896] Lustre: 24311:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 395.166096] Lustre: 24311:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 395.169438] Lustre: 24311:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 395.172379] Lustre: 24311:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 398.253776] Lustre: 25778:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 398.257930] Lustre: 25778:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 1 previous similar message [ 398.261808] Lustre: 25778:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 398.266573] Lustre: 25778:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 398.270311] Lustre: 25778:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 398.276454] Lustre: 25778:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 398.279661] Lustre: 25778:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 398.283905] Lustre: 25778:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 398.288339] Lustre: 25778:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 398.293222] Lustre: 25778:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 398.296784] Lustre: 25778:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 398.300377] Lustre: 25778:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 405.051500] Lustre: 24281:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 405.056031] Lustre: 24281:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 3 previous similar messages [ 405.061406] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 405.069662] Lustre: 24281:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 405.077678] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 405.081420] Lustre: 24281:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 405.084612] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 405.088433] Lustre: 24281:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 405.094131] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 405.100492] Lustre: 24281:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 405.104338] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 405.108576] Lustre: 24281:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 415.454408] Lustre: 24311:0:(osd_internal.h:1461:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 415.458567] Lustre: 24311:0:(osd_internal.h:1461:osd_trans_exec_op()) Skipped 5 previous similar messages [ 415.462281] Lustre: 24311:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 415.466331] Lustre: 24311:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 415.469713] Lustre: 24311:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 415.472657] Lustre: 24311:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 415.476464] Lustre: 24311:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 415.479906] Lustre: 24311:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 415.483395] Lustre: 24311:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 415.486532] Lustre: 24311:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 415.489577] Lustre: 24311:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 415.493039] Lustre: 24311:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 456.673489] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 456.674178] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 456.683767] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 461.796967] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 462.763630] Lustre: server umount lustre-OST07c6 complete [ 467.220431] Lustre: server umount lustre-MDT0000 complete [ 474.414562] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 475.713268] Key type lgssc unregistered [ 475.897688] LNet: 27796:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 475.905771] LNetError: 27796:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 475.916757] LNet: Removed LNI 192.168.206.140@tcp [ 476.319602] Key type .llcrypt unregistered [ 476.322113] Key type ._llcrypt unregistered [ 485.318342] Key type ._llcrypt registered [ 485.320310] Key type .llcrypt registered [ 485.363646] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 485.830603] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 485.840504] alg: No test for adler32 (adler32-zlib) [ 486.742337] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 486.856443] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 488.464197] Key type lgssc registered [ 488.964653] Lustre: Echo OBD driver; http://www.lustre.org/ [ 492.557369] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 492.573718] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 493.817684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 495.552516] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 499.748336] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 499.960895] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 501.793602] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 504.480378] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 506.104825] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 509.112694] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 509.296845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 512.096312] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 514.668249] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 518.508300] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 525.796432] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 525.802279] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 525.810546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 527.845377] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 527.845445] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 527.850504] LustreError: Skipped 1 previous similar message [ 527.859255] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.866135] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 528.871694] Lustre: Skipped 1 previous similar message [ 531.874208] Lustre: server umount lustre-OST0000 complete [ 533.985526] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 533.986805] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 533.995338] Lustre: Skipped 1 previous similar message [ 539.106054] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.109561] Lustre: Skipped 2 previous similar messages [ 539.823656] Lustre: server umount lustre-MDT0000 complete [ 541.566533] LustreError: 29010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219059 with bad export cookie 10982255155405856983 [ 541.569690] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 541.573861] LustreError: 29010:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 541.740774] Lustre: server umount lustre-MDT0001 complete [ 546.104287] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 547.373881] Key type lgssc unregistered [ 547.515475] LNet: 32405:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 547.519188] LNetError: 32405:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 547.530374] LNet: Removed LNI 192.168.206.140@tcp [ 547.936922] Key type .llcrypt unregistered [ 547.939112] Key type ._llcrypt unregistered [ 556.395745] Key type ._llcrypt registered [ 556.397300] Key type .llcrypt registered [ 556.448132] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 556.793483] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 556.804669] alg: No test for adler32 (adler32-zlib) [ 557.694365] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 557.806814] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 559.416181] Key type lgssc registered [ 559.889328] Lustre: Echo OBD driver; http://www.lustre.org/ [ 563.519072] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 563.534873] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 564.767031] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 566.329170] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 569.745097] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 569.923079] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 571.597294] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 574.135302] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 575.704168] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 579.045943] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 579.271169] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 582.182202] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 584.933867] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 589.549354] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 595.940054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 595.944074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 595.951163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 600.035768] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 600.036930] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 600.041182] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 601.840609] Lustre: server umount lustre-OST0000 complete [ 605.153827] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 605.162423] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 605.164532] Lustre: Skipped 2 previous similar messages [ 609.766057] Lustre: server umount lustre-MDT0000 complete [ 610.273648] LustreError: 33636:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 610.281814] LustreError: 33636:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 611.407411] LustreError: 33616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219129 with bad export cookie 7925562042369798697 [ 611.409134] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 611.413570] LustreError: 33616:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 611.544295] Lustre: server umount lustre-MDT0001 complete [ 615.908061] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 617.264864] Key type lgssc unregistered [ 617.413311] LNet: 36807:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 617.417236] LNetError: 36807:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 617.427594] LNet: Removed LNI 192.168.206.140@tcp [ 617.801792] Key type .llcrypt unregistered [ 617.803602] Key type ._llcrypt unregistered [ 628.909929] Key type ._llcrypt registered [ 628.911442] Key type .llcrypt registered [ 628.964377] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 629.431303] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 629.444418] alg: No test for adler32 (adler32-zlib) [ 630.365397] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 630.474966] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 632.080233] Key type lgssc registered [ 632.593198] Lustre: Echo OBD driver; http://www.lustre.org/ [ 636.307495] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 636.320285] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 637.549224] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 639.208567] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 642.986468] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 643.165067] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 644.766978] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 647.183145] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 648.885950] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 652.185549] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 652.345511] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 655.033332] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 657.385901] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 662.568684] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 667.618590] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 667.620095] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 667.625554] Lustre: Skipped 2 previous similar messages [ 667.627969] Lustre: Skipped 1 previous similar message [ 669.032416] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 669.035681] Lustre: Skipped 1 previous similar message [ 672.421143] Lustre: server umount lustre-MDT0000 complete [ 672.739932] LustreError: 38183:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 672.747987] LustreError: 38183:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 673.956532] LustreError: 38163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219192 with bad export cookie 16551813492690906410 [ 673.959288] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 673.964186] LustreError: 38163:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 674.145134] LustreError: 38180:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 674.155786] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.40@tcp (stopping) [ 677.860744] LustreError: 38178:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 677.862270] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 677.881841] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 680.127755] Lustre: server umount lustre-MDT0001 complete [ 698.528944] Lustre: server umount lustre-OST0000 complete [ 704.873151] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 706.177628] Key type lgssc unregistered [ 706.346457] LNet: 41685:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 706.351617] LNetError: 41685:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 706.360310] LNet: Removed LNI 192.168.206.140@tcp [ 706.751703] Key type .llcrypt unregistered [ 706.753736] Key type ._llcrypt unregistered [ 717.366792] Key type ._llcrypt registered [ 717.368942] Key type .llcrypt registered [ 717.425239] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 717.793355] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 717.858076] alg: No test for adler32 (adler32-zlib) [ 718.738623] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 718.840899] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 720.432227] Key type lgssc registered [ 720.942980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 724.642670] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 724.656982] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 725.923262] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 727.698335] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 731.276232] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 731.434521] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 733.070069] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 735.669758] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 737.404159] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 740.846693] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 741.038278] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 743.951621] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 746.719813] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 751.277845] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 756.707882] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 756.710024] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 756.715670] Lustre: Skipped 1 previous similar message [ 756.727502] Lustre: Skipped 1 previous similar message [ 758.127295] Lustre: lustre-OST0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 761.563231] Lustre: server umount lustre-OST0000 complete [ 775.876503] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 775.888476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 775.891359] Lustre: Skipped 1 previous similar message [ 782.089067] Lustre: server umount lustre-MDT0000 complete [ 782.306911] LustreError: 43061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 782.320989] LustreError: 43061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 783.984370] LustreError: 43046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219302 with bad export cookie 2446026595806274023 [ 783.990517] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 783.994074] LustreError: 43046:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 790.226403] Lustre: server umount lustre-MDT0001 complete [ 794.991993] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 796.302557] Key type lgssc unregistered [ 796.445623] LNet: 46410:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 796.451195] LNetError: 46410:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 796.463347] LNet: Removed LNI 192.168.206.140@tcp [ 796.815916] Key type .llcrypt unregistered [ 796.817541] Key type ._llcrypt unregistered [ 808.431761] Key type ._llcrypt registered [ 808.433789] Key type .llcrypt registered [ 808.487512] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 808.956976] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 809.016383] alg: No test for adler32 (adler32-zlib) [ 809.963308] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 810.101679] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 811.704165] Key type lgssc registered [ 812.247273] Lustre: Echo OBD driver; http://www.lustre.org/ [ 816.212352] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 816.224497] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 817.488056] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 819.583966] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 823.916879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 824.131316] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 826.094668] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 828.865876] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 830.536419] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 833.643762] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 833.819236] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 836.420508] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 839.007904] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 842.025254] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 847.794712] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 849.342040] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 860.331621] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-06-1119h09m38s [ 861.040410] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 862.689388] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 862.691786] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 862.696129] Lustre: Skipped 2 previous similar messages [ 862.698515] Lustre: Skipped 1 previous similar message [ 863.715159] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 863.719840] Lustre: Skipped 2 previous similar messages [ 866.155904] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 866.162552] Lustre: Skipped 1 previous similar message [ 868.135614] Lustre: server umount lustre-MDT0000 complete [ 869.562960] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 869.858036] LustreError: 47788:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 869.865185] LustreError: 47788:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 872.287645] LustreError: 47785:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 872.515426] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 872.589355] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 872.737676] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 872.776182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 873.766197] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 874.486394] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 876.013157] Lustre: DEBUG MARKER: Wait for df (35955) ... [ 876.662306] Lustre: DEBUG MARKER: done [ 877.181628] LustreError: 49405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.40@tcp arrived at 1781219395 with bad export cookie 2916643395353343512 [ 877.188879] LustreError: 49405:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 879.076810] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 879.077533] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 879.082400] Lustre: Skipped 1 previous similar message [ 879.085095] Lustre: Skipped 1 previous similar message [ 884.194831] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 884.199466] Lustre: Skipped 2 previous similar messages [ 892.386338] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 892.470352] Lustre: server umount lustre-OST0000 complete [ 894.434557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 894.435922] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 894.438166] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 894.438175] Lustre: Skipped 2 previous similar messages [ 894.452845] Lustre: Skipped 1 previous similar message [ 900.412666] Lustre: server umount lustre-MDT0000 complete [ 900.577745] LustreError: 47789:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 900.585747] LustreError: 47789:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 902.256402] LustreError: 47768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219420 with bad export cookie 2916643395353343953 [ 902.261489] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 902.264055] LustreError: 47768:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 902.425036] Lustre: server umount lustre-MDT0001 complete [ 907.456845] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 908.776350] Key type lgssc unregistered [ 908.932224] LNet: 52332:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 908.939478] LNetError: 52332:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 908.949164] LNet: Removed LNI 192.168.206.140@tcp [ 909.368157] Key type .llcrypt unregistered [ 909.370206] Key type ._llcrypt unregistered [ 929.128392] Key type ._llcrypt registered [ 929.129889] Key type .llcrypt registered [ 929.179971] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 929.588935] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 929.649217] alg: No test for adler32 (adler32-zlib) [ 930.549389] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 930.684860] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 932.280274] Key type lgssc registered [ 932.750173] Lustre: Echo OBD driver; http://www.lustre.org/ [ 936.127915] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 936.136067] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 937.286591] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 937.303989] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 937.389869] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 939.086463] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 943.025776] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 943.084469] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 943.098645] Lustre: 54498:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 943.218162] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 944.843195] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 947.199695] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 948.730493] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 951.540173] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 951.629324] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 951.719280] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 953.998110] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 956.380846] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 956.965078] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 965.415476] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-06-1119h11m23s [ 966.088906] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 969.190262] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 970.306603] Lustre: DEBUG MARKER: Creating a test file and stat it [ 970.324980] LustreError: 54507:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 990.579067] Lustre: lustre-MDT0000: Client 4a2c6ddd-44f0-4698-b049-e62e274dbc74 (at 192.168.206.40@tcp) reconnecting [ 1010.368567] LustreError: 54507:0:(mdt_handler.c:2347:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1011.514984] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1012.930450] Lustre: DEBUG MARKER: done [ 1015.781868] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1015.790206] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1015.798913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1018.338196] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1018.340098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1018.344293] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1018.352596] Lustre: Skipped 1 previous similar message [ 1021.542737] Lustre: server umount lustre-OST0000 complete [ 1024.993334] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1024.996996] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1025.007061] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.090096] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1029.096629] Lustre: Skipped 4 previous similar messages [ 1029.752645] Lustre: server umount lustre-MDT0000 complete [ 1031.613479] LustreError: 54490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219549 with bad export cookie 13079572624335886555 [ 1031.616789] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1031.619766] LustreError: 54490:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1031.795424] Lustre: server umount lustre-MDT0001 complete [ 1036.509260] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1037.935674] Key type lgssc unregistered [ 1038.115428] LNet: 58466:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1038.121605] LNetError: 58466:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1038.134808] LNet: Removed LNI 192.168.206.140@tcp [ 1038.548977] Key type .llcrypt unregistered [ 1038.551516] Key type ._llcrypt unregistered [ 1059.919721] Key type ._llcrypt registered [ 1059.923953] Key type .llcrypt registered [ 1059.997384] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1060.434968] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1060.494625] alg: No test for adler32 (adler32-zlib) [ 1061.426570] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1061.552916] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1063.144136] Key type lgssc registered [ 1063.651257] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1065.450927] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1065.461251] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1066.613594] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1066.629290] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1066.719649] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1067.205562] Lustre: Failing over lustre-MDT0000 [ 1067.351212] Lustre: server umount lustre-MDT0000 complete [ 1070.622959] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 19:13:08 (1781219588) [ 1074.339188] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1074.650573] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1076.436459] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1080.988826] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1081.068855] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1081.089677] Lustre: 61221:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1081.224817] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1083.488476] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1086.255144] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1087.954966] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1091.414135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1091.524664] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1091.627910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1094.562356] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1097.314243] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1098.802085] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1107.034292] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1112.033215] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1112.038917] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1112.049270] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.082266] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1114.084818] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1114.086983] Lustre: Skipped 1 previous similar message [ 1114.094871] Lustre: Skipped 1 previous similar message [ 1116.405201] Lustre: server umount lustre-MDT0000 complete [ 1118.159289] LustreError: 62849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219636 with bad export cookie 3397104667816779990 [ 1118.162048] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1118.164609] LustreError: 62849:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1118.308074] Lustre: server umount lustre-MDT0001 complete [ 1119.896740] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1122.236369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1126.754520] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1128.354477] LustreError: 60145:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000134eea10 x1867744101303296/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1128.556566] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.614478] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1130.327644] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1134.053161] LustreError: 62841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1134.054380] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1134.061505] LustreError: 62841:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1134.226947] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1135.392715] Lustre: 60149:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219637/real 1781219637] req@0000000024588674 x1867744101302912/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781219653 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1135.400426] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1136.156737] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1138.687194] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1139.488103] Lustre: 60146:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219641/real 1781219641] req@0000000040337236 x1867744101303040/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781219657 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1140.424180] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1140.711092] Lustre: 60147:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219642/real 1781219642] req@00000000c1459232 x1867744101303168/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781219658 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1143.904291] Lustre: 60148:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219646/real 1781219646] req@00000000a04bd479 x1867744101303424/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781219662 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1144.801438] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1144.805406] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1144.812333] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1144.815360] Lustre: Skipped 1 previous similar message [ 1149.025893] Lustre: server umount lustre-MDT0000 complete [ 1150.618653] LustreError: 62848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219668 with bad export cookie 3397104667816782258 [ 1150.622746] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1150.628206] LustreError: 62848:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1150.765974] Lustre: server umount lustre-MDT0001 complete [ 1153.735377] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1158.178704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1160.097087] LustreError: 60145:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000005e08491e x1867744101323392/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1160.252863] LustreError: 62840:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1160.264403] LustreError: 62840:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1160.312156] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1160.315285] Lustre: Skipped 1 previous similar message [ 1160.349262] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1162.321439] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1165.793930] LustreError: 62841:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1165.794505] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1165.811038] Lustre: Skipped 2 previous similar messages [ 1166.549460] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1167.712125] Lustre: 60149:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219669/real 1781219669] req@0000000023c8793c x1867744101323136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781219685 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1167.787834] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1167.791447] Lustre: Skipped 1 previous similar message [ 1168.537201] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1171.174337] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1172.814443] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1176.352126] Lustre: 60146:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781219678/real 1781219678] req@00000000922c8389 x1867744101323520/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781219694 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1176.364507] Lustre: 60146:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 1177.057539] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1177.063071] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1177.071164] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1177.075765] Lustre: Skipped 3 previous similar messages [ 1181.490419] Lustre: server umount lustre-MDT0000 complete [ 1183.201974] LustreError: 67098:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1183.211323] LustreError: 67098:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1183.255728] LustreError: 61215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219701 with bad export cookie 3397104667816783637 [ 1183.256037] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1183.264510] LustreError: 61215:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 1183.419724] Lustre: server umount lustre-MDT0001 complete [ 1186.519615] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1188.366537] Lustre: DEBUG MARKER: files compared the same [ 1199.806471] Lustre: server umount lustre-OST0000 complete [ 1207.149836] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1208.566943] Key type lgssc unregistered [ 1208.721877] LNet: 70125:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1208.726249] LNetError: 70125:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1208.739088] LNet: Removed LNI 192.168.206.140@tcp [ 1209.143266] Key type .llcrypt unregistered [ 1209.145112] Key type ._llcrypt unregistered [ 1220.437292] Key type ._llcrypt registered [ 1220.438718] Key type .llcrypt registered [ 1220.492876] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1221.050979] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1221.095243] alg: No test for adler32 (adler32-zlib) [ 1222.015852] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1222.142075] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1223.744168] Key type lgssc registered [ 1224.332661] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1228.188472] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1228.197856] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1229.442667] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1231.304270] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug +malloc all [ 1235.166575] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1235.308946] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1237.038882] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug +malloc all [ 1239.240543] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1240.775591] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1243.643580] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1243.772652] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1246.025235] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug +malloc all [ 1248.498968] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1253.989842] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1259.489737] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1259.490457] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1259.490902] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1259.496938] Lustre: Skipped 1 previous similar message [ 1263.711051] Lustre: server umount lustre-OST0000 complete [ 1266.144777] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1266.148860] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1266.157756] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1266.160377] Lustre: Skipped 1 previous similar message [ 1269.728854] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1269.733859] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1269.737723] Lustre: Skipped 1 previous similar message [ 1271.474313] Lustre: server umount lustre-MDT0000 complete [ 1273.153070] LustreError: 73936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219791 with bad export cookie 7622839374567679876 [ 1273.154036] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1273.161413] LustreError: 73936:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1273.341324] Lustre: server umount lustre-MDT0001 complete [ 1278.146616] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1279.487935] Key type lgssc unregistered [ 1279.634374] LNet: 74670:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1279.637442] LNetError: 74670:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1279.646560] LNet: Removed LNI 192.168.206.140@tcp [ 1279.968160] Key type .llcrypt unregistered [ 1279.970184] Key type ._llcrypt unregistered [ 1290.795113] Key type ._llcrypt registered [ 1290.796909] Key type .llcrypt registered [ 1290.852943] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1291.442784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1291.454463] alg: No test for adler32 (adler32-zlib) [ 1292.357517] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1292.495884] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1294.096216] Key type lgssc registered [ 1294.641462] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1298.185760] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1298.198346] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1313.632342] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1318.752230] LustreError: 76021:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.140@tcp: failed processing log, type 4: rc = -110 [ 1345.376188] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1347.600576] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1349.656642] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1352.860185] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1352.929133] Lustre: *** cfs_fail_loc=706, val=0*** [ 1353.095228] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1353.125624] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1354.924558] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1358.307068] LustreError: 76047:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1358.314025] LustreError: 76047:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1358.539101] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1358.697367] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1360.391430] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1362.600431] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1364.076440] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1365.990689] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1366.000288] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1366.003719] Lustre: Skipped 1 previous similar message [ 1371.107921] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1371.110870] Lustre: Skipped 1 previous similar message [ 1376.230863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1376.233585] Lustre: Skipped 1 previous similar message [ 1379.296145] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1379.362683] Lustre: server umount lustre-OST0000 complete [ 1381.345402] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1381.347247] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1381.351966] Lustre: Skipped 2 previous similar messages [ 1381.357063] Lustre: Skipped 1 previous similar message [ 1386.465971] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1386.470184] Lustre: Skipped 1 previous similar message [ 1387.177405] Lustre: server umount lustre-MDT0000 complete [ 1388.920766] LustreError: 78338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781219907 with bad export cookie 14772321149538347078 [ 1388.923511] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1388.924750] LustreError: 78338:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1389.068643] Lustre: server umount lustre-MDT0001 complete [ 1393.198346] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1394.439635] Key type lgssc unregistered [ 1394.598185] LNet: 79277:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1394.605049] LNetError: 79277:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1394.617715] LNet: Removed LNI 192.168.206.140@tcp [ 1394.985951] Key type .llcrypt unregistered [ 1394.987226] Key type ._llcrypt unregistered [ 1405.654053] Key type ._llcrypt registered [ 1405.655926] Key type .llcrypt registered [ 1405.700835] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1406.050636] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1406.062262] alg: No test for adler32 (adler32-zlib) [ 1406.922838] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1407.029503] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1408.616232] Key type lgssc registered [ 1409.083229] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1412.544645] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1412.552732] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1415.297738] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1418.311490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1418.562567] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1420.210433] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1423.774945] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1423.841830] LustreError: 81339:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1423.849963] LustreError: 81339:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1423.944400] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1426.551043] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1428.065932] LustreError: 81342:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1428.074835] LustreError: 81342:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 1430.425580] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1430.433352] Lustre: MGS: Client cf71a1ed-c0fb-43c3-a8c8-2be44a404e74 (at 0@lo) reconnecting [ 1430.442687] LustreError: 81353:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1430.551764] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.588214] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1432.227055] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1445.863218] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1445.863753] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1445.868424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1445.870514] Lustre: Skipped 1 previous similar message [ 1448.189276] Lustre: server umount lustre-OST0000 complete [ 1450.978263] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1450.979346] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1450.981253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1450.981260] Lustre: Skipped 1 previous similar message [ 1450.999265] Lustre: Skipped 1 previous similar message [ 1456.098334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1456.101601] Lustre: Skipped 2 previous similar messages [ 1461.218791] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1461.222408] Lustre: Skipped 1 previous similar message [ 1464.288207] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1464.345256] Lustre: server umount lustre-MDT0000 complete [ 1466.166878] Lustre: server umount lustre-MDT0001 complete [ 1467.908432] Lustre: server umount MGS complete [ 1472.661992] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 19:19:50 (1781219990) [ 1485.756499] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 1489.468364] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1505.414682] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1507.212061] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1512.166411] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1512.273577] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1512.297200] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1512.342453] Lustre: lustre-MDT0001: new disk, initializing [ 1512.387597] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1512.406958] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1512.413420] Lustre: 88399:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1512.418439] LustreError: 88399:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1514.231834] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1519.467738] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1519.588933] Lustre: lustre-OST0000: new disk, initializing [ 1519.591787] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1519.594966] Lustre: 89147:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1519.627247] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1521.872724] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1522.721365] Lustre: 89433:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000bda814d6 x1867744464112256/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781220056 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1525.322824] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1528.275689] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1528.280888] Lustre: MGS: Client ff28fe35-dac6-4a1d-ac1c-a386c8656df6 (at 0@lo) reconnecting [ 1528.303839] Lustre: 87687:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1528.312290] LustreError: 88385:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1528.316602] LustreError: 88385:0:(mgs_handler.c:244:mgs_revoke_lock()) Skipped 1 previous similar message [ 1528.331925] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1528.372280] Lustre: lustre-MDT0000: new disk, initializing [ 1528.464760] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1528.480782] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1529.505054] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1529.511644] Lustre: 90103:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000882f3ecf x1867744464118272/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1781220063 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1530.531930] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1531.561603] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1531.565968] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1531.635116] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1551.336078] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1551.338193] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1551.341997] Lustre: Skipped 1 previous similar message [ 1556.523187] Lustre: server umount lustre-OST0000 complete [ 1561.570208] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1561.570494] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1561.573539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1561.579681] Lustre: Skipped 1 previous similar message [ 1561.582573] Lustre: Skipped 3 previous similar messages [ 1564.427680] Lustre: server umount lustre-MDT0000 complete [ 1572.333247] Lustre: server umount lustre-MDT0001 complete [ 1574.359845] Lustre: server umount MGS complete [ 1579.632072] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 19:21:37 (1781220097) [ 1587.713320] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1589.025214] Key type lgssc unregistered [ 1589.176058] LNet: 92565:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1589.180457] LNetError: 92565:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1589.190451] LNet: Removed LNI 192.168.206.140@tcp [ 1589.608370] Key type .llcrypt unregistered [ 1589.610376] Key type ._llcrypt unregistered [ 1597.461948] Key type ._llcrypt registered [ 1597.463583] Key type .llcrypt registered [ 1597.515392] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1597.936506] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1597.971097] alg: No test for adler32 (adler32-zlib) [ 1598.846894] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1598.958230] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1600.552164] Key type lgssc registered [ 1601.017238] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1604.753842] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1604.772419] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1606.051595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1607.997583] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1611.878696] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1612.060376] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1613.756073] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1616.285449] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1617.951822] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1621.203214] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1621.377549] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1624.319315] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1627.031164] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1631.662517] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1636.836823] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1636.837440] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1636.842182] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1636.845898] Lustre: Skipped 1 previous similar message [ 1636.850050] Lustre: Skipped 1 previous similar message [ 1641.365746] Lustre: server umount lustre-OST0000 complete [ 1642.977367] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1642.980977] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1642.990531] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.076450] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1647.077936] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1652.194769] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1652.198295] Lustre: Skipped 2 previous similar messages [ 1657.312416] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1657.313620] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1657.322795] Lustre: Skipped 1 previous similar message [ 1657.409535] Lustre: server umount lustre-MDT0000 complete [ 1659.102227] LustreError: 93726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220177 with bad export cookie 9742420132141207163 [ 1659.104855] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1659.107952] LustreError: 93726:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 1659.264324] Lustre: server umount lustre-MDT0001 complete [ 1663.806812] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1665.161039] Key type lgssc unregistered [ 1665.324663] LNet: 96921:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1665.331554] LNetError: 96921:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1665.351870] LNet: Removed LNI 192.168.206.140@tcp [ 1665.786751] Key type .llcrypt unregistered [ 1665.788363] Key type ._llcrypt unregistered [ 1673.430665] Key type ._llcrypt registered [ 1673.432304] Key type .llcrypt registered [ 1673.487332] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1673.887460] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1673.932092] alg: No test for adler32 (adler32-zlib) [ 1674.825997] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1674.942279] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1676.560585] Key type lgssc registered [ 1677.030486] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1679.700187] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1679.703184] LustreError: 97967:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1679.704175] Lustre: Skipped 1 previous similar message [ 1679.710656] LustreError: 97965:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1679.714444] LustreError: 97967:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4994 [ 1679.717597] LustreError: 97967:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 1679.722878] LustreError: 97967:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1679.725167] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1680.990860] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1684.316502] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1684.477191] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1686.238218] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1688.633048] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing lsmod [ 1689.958180] LustreError: 99138:0:(tgt_mount.c:2522:server_fill_super()) cfs_race id 716 sleeping [ 1689.958192] LustreError: 99137:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 waking [ 1689.965829] LustreError: 99138:0:(tgt_mount.c:2522:server_fill_super()) cfs_fail_race id 716 awake: rc=4996 [ 1689.971066] LustreError: 99138:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 1689.976900] LustreError: 99138:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 1689.989995] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1690.192891] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1691.246970] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 1694.691600] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1694.694165] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1694.698755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1694.719758] Lustre: Skipped 2 previous similar messages [ 1697.780848] Lustre: server umount lustre-MDT0000 complete [ 1699.649476] LustreError: 97985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220217 with bad export cookie 8393546047642390682 [ 1699.652652] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1699.654927] LustreError: 97985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1699.810991] LustreError: 98001:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1699.811910] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1699.823536] LustreError: 98001:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 1699.831301] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 1699.841106] Lustre: Skipped 2 previous similar messages [ 1700.896337] LustreError: 98435:0:(client.c:1380:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@000000007bbcf51f x1867744745113600/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1700.985158] Lustre: server umount lustre-MDT0001 complete [ 1702.764935] Lustre: server umount lustre-OST0000 complete [ 1706.773326] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1706.988233] LustreError: 100200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1707.020595] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1709.104603] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1712.099494] LustreError: 100200:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1712.107858] LustreError: 100200:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 1713.902219] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1714.125261] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1716.558532] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1719.449175] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1721.078758] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1724.502743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1724.683377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1726.999243] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1729.329911] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1731.883653] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:97) [ 1744.359062] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1744.359630] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1744.364697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1744.370944] Lustre: Skipped 1 previous similar message [ 1744.375954] Lustre: Skipped 1 previous similar message [ 1748.834405] Lustre: server umount lustre-OST0000 complete [ 1754.599608] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1754.605189] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1754.606027] Lustre: Skipped 1 previous similar message [ 1754.614648] Lustre: Skipped 1 previous similar message [ 1756.731973] Lustre: server umount lustre-MDT0000 complete [ 1758.417782] LustreError: 100181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220276 with bad export cookie 8393546047642391599 [ 1758.418965] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1758.425317] LustreError: 100181:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1758.601095] Lustre: server umount lustre-MDT0001 complete [ 1762.992590] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1764.320931] Key type lgssc unregistered [ 1764.468549] LNet: 103400:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1764.471860] LNetError: 103400:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1764.482359] LNet: Removed LNI 192.168.206.140@tcp [ 1764.855534] Key type .llcrypt unregistered [ 1764.860938] Key type ._llcrypt unregistered [ 1775.577134] Key type ._llcrypt registered [ 1775.578826] Key type .llcrypt registered [ 1775.633501] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1776.236323] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1776.255436] alg: No test for adler32 (adler32-zlib) [ 1777.151852] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1777.278748] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1778.880230] Key type lgssc registered [ 1779.387694] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1782.751914] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1782.767447] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1784.045939] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1785.758760] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1789.204569] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1789.385423] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1791.078778] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1793.426206] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1794.880121] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1797.804270] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1797.963594] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1800.264334] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1802.648697] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1809.260467] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 1815.642157] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 1816.784688] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 1819.621749] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1819.623223] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1819.624494] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1819.628687] Lustre: Skipped 1 previous similar message [ 1819.634728] LustreError: Skipped 1 previous similar message [ 1824.382169] Lustre: server umount lustre-OST0000 complete [ 1829.859830] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1829.860991] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1829.866250] Lustre: Skipped 1 previous similar message [ 1829.869238] Lustre: Skipped 1 previous similar message [ 1832.476666] Lustre: server umount lustre-MDT0000 complete [ 1834.400545] LustreError: 104758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220352 with bad export cookie 2495454290527406693 [ 1834.403401] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1834.405738] LustreError: 104758:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1834.551709] Lustre: server umount lustre-MDT0001 complete [ 1839.528056] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1840.874392] Key type lgssc unregistered [ 1841.022650] LNet: 108074:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1841.027094] LNetError: 108074:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1841.035520] LNet: Removed LNI 192.168.206.140@tcp [ 1841.400932] Key type .llcrypt unregistered [ 1841.402700] Key type ._llcrypt unregistered [ 1849.781152] Key type ._llcrypt registered [ 1849.783271] Key type .llcrypt registered [ 1849.846588] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1850.344211] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1850.388090] alg: No test for adler32 (adler32-zlib) [ 1851.263594] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1851.361477] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1852.952584] Key type lgssc registered [ 1853.508646] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1857.239665] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1857.250726] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1858.522830] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1860.581462] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1864.819799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1865.010956] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1866.873850] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1869.449771] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1871.172902] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1874.304590] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1874.457971] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1877.025114] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1879.298223] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1883.689592] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 1890.785613] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1890.789508] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1890.798036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1893.858354] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1893.862188] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1893.868598] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1895.275911] Lustre: server umount lustre-OST0000 complete [ 1898.979038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1898.980476] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1898.987574] Lustre: Skipped 1 previous similar message [ 1898.990385] Lustre: Skipped 1 previous similar message [ 1904.097327] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1904.100924] Lustre: Skipped 2 previous similar messages [ 1909.217424] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1909.222022] Lustre: Skipped 1 previous similar message [ 1911.264175] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1911.351333] Lustre: server umount lustre-MDT0000 complete [ 1913.062982] LustreError: 109237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220431 with bad export cookie 448956977090159787 [ 1913.064289] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1913.069918] LustreError: 109237:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1913.226036] Lustre: server umount lustre-MDT0001 complete [ 1918.933620] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 1920.420392] Key type lgssc unregistered [ 1920.592961] LNet: 112433:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1920.596356] LNetError: 112433:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1921.638733] LNet: Removed LNI 192.168.206.140@tcp [ 1922.098592] Key type .llcrypt unregistered [ 1922.100271] Key type ._llcrypt unregistered [ 1933.999169] Key type ._llcrypt registered [ 1934.001298] Key type .llcrypt registered [ 1934.073071] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 1934.559938] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1934.605696] alg: No test for adler32 (adler32-zlib) [ 1935.516174] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 1935.657780] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 1937.280241] Key type lgssc registered [ 1937.749980] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1941.566375] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1941.578854] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1942.802450] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1944.650397] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1948.650319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1948.856458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1950.790627] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1953.537259] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 1955.256627] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 1958.465313] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1958.666643] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1961.348674] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 1963.743084] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 1966.890328] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:163 to 0x280000400:193) [ 1976.924405] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 1978.819112] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 1978.822833] Lustre: Skipped 2 previous similar messages [ 1981.216378] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 1981.219136] Lustre: Skipped 2 previous similar messages [ 1984.009849] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 1984.033269] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 1987.629149] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.40@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 1987.634526] Lustre: Skipped 2 previous similar messages [ 1990.991713] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.40@tcp 192.168.0.[2,10]@tcp [ 1994.914975] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.40/24@tcp in log lustre-MDT0000 [ 1994.918477] Lustre: Skipped 2 previous similar messages [ 1997.579862] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.40/24@tcp [ 1997.582633] Lustre: Skipped 1 previous similar message [ 2002.913468] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2002.917691] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2002.922567] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2003.940115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2003.940477] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2003.940995] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2007.045265] Lustre: server umount lustre-OST0000 complete [ 2009.058769] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.059901] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2009.069214] Lustre: Skipped 1 previous similar message [ 2009.071390] Lustre: Skipped 1 previous similar message [ 2014.177187] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2014.180538] Lustre: Skipped 2 previous similar messages [ 2019.298722] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2019.302806] Lustre: Skipped 1 previous similar message [ 2023.392191] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2023.518112] Lustre: server umount lustre-MDT0000 complete [ 2024.424385] LustreError: 113811:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2024.439665] LustreError: 113811:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 2025.733058] LustreError: 115424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220543 with bad export cookie 17601415383566079610 [ 2025.734159] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.739415] LustreError: 115424:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2025.947289] Lustre: server umount lustre-MDT0001 complete [ 2031.372371] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2032.384118] hrtimer: interrupt took 4549772 ns [ 2033.041466] Key type lgssc unregistered [ 2033.221427] LNet: 118356:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2033.226133] LNetError: 118356:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2033.245117] LNet: Removed LNI 192.168.206.140@tcp [ 2033.754641] Key type .llcrypt unregistered [ 2033.756905] Key type ._llcrypt unregistered [ 2048.731532] Key type ._llcrypt registered [ 2048.733418] Key type .llcrypt registered [ 2048.794701] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2049.228767] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2049.295123] alg: No test for adler32 (adler32-zlib) [ 2050.253798] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2050.463537] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2052.096340] Key type lgssc registered [ 2052.791377] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2057.310902] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2057.333638] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2058.620153] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2058.639612] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.40/24@tcp [ 2058.640369] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2060.760299] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2065.113927] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2065.330163] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2065.345796] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.40/24@tcp [ 2065.363458] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2067.199748] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2069.774142] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2071.663478] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2074.897129] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2075.094728] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2078.158449] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2080.935460] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2083.320107] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:201 to 0x280000400:225) [ 2091.489020] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2091.492613] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2091.502310] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2093.538836] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2093.541752] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2093.542399] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2097.094680] Lustre: server umount lustre-OST0000 complete [ 2101.221764] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2101.225444] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2101.233947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2101.239351] Lustre: Skipped 1 previous similar message [ 2103.777939] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2103.778435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2103.792803] Lustre: Skipped 1 previous similar message [ 2105.268454] Lustre: server umount lustre-MDT0000 complete [ 2106.844445] LustreError: 119856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220624 with bad export cookie 16310323556950741393 [ 2106.849356] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2106.850734] LustreError: 119856:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2106.974543] Lustre: server umount lustre-MDT0001 complete [ 2111.356120] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2112.625960] Key type lgssc unregistered [ 2112.785894] LNet: 123104:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2112.789672] LNetError: 123104:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2112.801592] LNet: Removed LNI 192.168.206.140@tcp [ 2113.174476] Key type .llcrypt unregistered [ 2113.176237] Key type ._llcrypt unregistered [ 2124.905814] Key type ._llcrypt registered [ 2124.907408] Key type .llcrypt registered [ 2124.963913] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 2133.155195] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2133.697581] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2133.718582] alg: No test for adler32 (adler32-zlib) [ 2134.682358] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2134.821132] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2136.456387] Key type lgssc registered [ 2137.197446] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2158.197647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2158.228240] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2159.369269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2159.393885] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2159.443543] Lustre: lustre-MDT0000: new disk, initializing [ 2159.495909] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2159.507225] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2161.986949] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2168.872445] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2168.977728] Lustre: 126447:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2169.035515] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2169.041111] Lustre: Skipped 1 previous similar message [ 2169.140793] Lustre: lustre-MDT0001: new disk, initializing [ 2169.232824] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2169.272391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2169.284192] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2171.771602] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2176.531504] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2178.552332] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2182.373263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2182.534889] Lustre: lustre-OST0000: new disk, initializing [ 2182.538958] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2182.544322] Lustre: 128185:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2182.601433] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2185.364894] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2185.372259] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2185.437145] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2185.980725] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2190.751301] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2193.582270] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2193.712165] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2196.221788] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2196.385678] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2201.060552] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2201.063475] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2201.067412] LustreError: Skipped 1 previous similar message [ 2201.068546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2201.077579] Lustre: Skipped 1 previous similar message [ 2203.655731] Lustre: server umount lustre-OST0000 complete [ 2206.178103] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2206.191513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2206.198824] Lustre: Skipped 2 previous similar messages [ 2211.298518] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2211.303445] Lustre: Skipped 1 previous similar message [ 2212.303308] Lustre: server umount lustre-MDT0000 complete [ 2214.491537] LustreError: 128184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220732 with bad export cookie 11477727182063453477 [ 2214.493322] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2214.497312] LustreError: 128184:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2214.716521] Lustre: server umount lustre-MDT0001 complete [ 2219.874502] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2220.183428] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2222.699492] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2225.637417] LustreError: 130252:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2225.652493] LustreError: 130252:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2227.883150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2228.146241] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2230.509284] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2233.801793] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2235.859612] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2239.846475] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2240.020793] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2243.028015] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2246.047125] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2248.513269] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2248.639618] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2250.550911] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2253.764840] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 3 sec [ 2258.367640] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2258.485454] Lustre: lustre-OST0001: new disk, initializing [ 2258.488335] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 2258.493161] Lustre: 133268:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2258.556670] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 2261.713538] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2266.739369] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 2267.172095] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 2267.176909] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 2267.245793] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 2267.904671] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 2270.137664] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 2270.263089] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2272.290517] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e6e07e56000.ost_server_uuid 50 [ 2273.064800] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e6e07e56000.ost_server_uuid in FULL state after 0 sec [ 2277.350356] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2277.350379] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2277.355257] LustreError: Skipped 1 previous similar message [ 2277.360696] Lustre: Skipped 1 previous similar message [ 2277.364707] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2277.368379] Lustre: Skipped 1 previous similar message [ 2282.468711] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 2282.471809] Lustre: Skipped 1 previous similar message [ 2282.893817] Lustre: server umount lustre-OST0001 complete [ 2286.050815] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2286.057779] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2287.590230] LustreError: 133260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2287.601170] LustreError: 133260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2292.710561] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2292.711558] LustreError: 131845:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2292.714373] Lustre: Skipped 4 previous similar messages [ 2292.722635] LustreError: 131845:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2297.830342] LustreError: 133260:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2297.842411] LustreError: 133260:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2299.360221] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2299.452763] Lustre: server umount lustre-OST0000 complete [ 2302.945317] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2302.955890] Lustre: Skipped 2 previous similar messages [ 2307.593354] Lustre: server umount lustre-MDT0000 complete [ 2308.066534] LustreError: 130963:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2308.082246] LustreError: 130963:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2309.753464] LustreError: 130232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220827 with bad export cookie 11477727182063454359 [ 2309.753923] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2309.758764] LustreError: 130232:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2309.926885] Lustre: server umount lustre-MDT0001 complete [ 2319.275279] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2320.583182] Key type lgssc unregistered [ 2320.728476] LNet: 136109:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2320.732218] LNetError: 136109:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2320.740840] LNet: Removed LNI 192.168.206.140@tcp [ 2321.139672] Key type .llcrypt unregistered [ 2321.141238] Key type ._llcrypt unregistered [ 2339.954891] Key type ._llcrypt registered [ 2339.956342] Key type .llcrypt registered [ 2340.023454] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 2347.894499] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2348.386750] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2348.406544] alg: No test for adler32 (adler32-zlib) [ 2349.297546] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2349.398432] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2351.000157] Key type lgssc registered [ 2351.517551] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2367.613079] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2367.621389] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2368.796945] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2368.816382] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2368.879315] Lustre: lustre-MDT0000: new disk, initializing [ 2368.932118] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2368.943788] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2371.257942] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2377.285595] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2377.339622] Lustre: 140217:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2377.371439] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2377.375083] Lustre: Skipped 1 previous similar message [ 2377.420352] Lustre: lustre-MDT0001: new disk, initializing [ 2377.474361] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2377.494760] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2377.500270] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2379.424873] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2383.951692] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2386.203871] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2390.009183] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2390.164911] Lustre: lustre-OST0000: new disk, initializing [ 2390.169792] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2390.175704] Lustre: 141966:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2390.240775] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2393.273693] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2394.638249] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2394.645248] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2394.701516] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2397.737620] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2401.455916] Lustre: Failing over lustre-OST0000 [ 2401.539754] Lustre: server umount lustre-OST0000 complete [ 2415.472902] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2415.643201] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2415.651899] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 2416.624099] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 2418.065140] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2419.692419] Lustre: Failing over lustre-MDT0000 [ 2419.792783] Lustre: server umount lustre-MDT0000 complete [ 2420.704161] Lustre: 138446:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781220922/real 1781220922] req@000000006dad6743 x1867745451864192/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781220938 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2420.704233] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2420.716037] Lustre: 138446:0:(client.c:2480:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 2420.721320] Lustre: Skipped 2 previous similar messages [ 2425.825318] LustreError: 140227:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.837266] LustreError: 140227:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 8 previous similar messages [ 2426.720612] LustreError: 142408:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2426.848424] Lustre: 138444:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781220928/real 1781220928] req@000000000fbbccbe x1867745451865472/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781220944 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2430.944262] Lustre: 138445:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781220933/real 1781220933] req@00000000bcb855c8 x1867745451866624/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1781220949 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2430.945192] LustreError: 140228:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2430.963439] LustreError: 140228:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2433.558402] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2433.649421] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2433.657216] Lustre: 140219:0:(mgc_request.c:1901:mgc_process_log()) MGC192.168.206.140@tcp: IR log lustre-mdtir failed, not fatal: rc = -5 [ 2433.755897] LustreError: lustre-OST0000-osc-MDT0000: operation ost_connect to node 0@lo failed: rc = -114 [ 2433.776788] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2433.814668] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 2435.442989] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2436.962881] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 2439.145167] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 2439.159136] Lustre: lustre-MDT0000: Recovery over after 0:03, of 2 clients 2 recovered and 0 were evicted. [ 2439.182226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 2444.259601] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2444.261525] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2444.266088] LustreError: Skipped 1 previous similar message [ 2444.268367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2444.275310] Lustre: Skipped 2 previous similar messages [ 2444.283197] Lustre: Skipped 1 previous similar message [ 2446.830434] Lustre: server umount lustre-OST0000 complete [ 2449.378692] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2449.378950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2449.384987] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2449.391957] Lustre: Skipped 1 previous similar message [ 2454.498383] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2454.501104] Lustre: Skipped 2 previous similar messages [ 2454.838588] Lustre: server umount lustre-MDT0000 complete [ 2456.672301] LustreError: 144716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781220974 with bad export cookie 12864122431864069769 [ 2456.677275] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2456.682308] LustreError: 144716:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2456.865938] Lustre: server umount lustre-MDT0001 complete [ 2462.464976] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2463.913459] Key type lgssc unregistered [ 2464.074954] LNet: 145450:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2464.079478] LNetError: 145450:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2464.092959] LNet: Removed LNI 192.168.206.140@tcp [ 2464.527165] Key type .llcrypt unregistered [ 2464.528965] Key type ._llcrypt unregistered [ 2490.419513] Key type ._llcrypt registered [ 2490.421136] Key type .llcrypt registered [ 2490.495513] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2490.995359] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2491.009792] alg: No test for adler32 (adler32-zlib) [ 2491.910954] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2492.008886] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2493.608635] Key type lgssc registered [ 2494.264212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2498.708910] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2498.720862] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2499.982323] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2501.825884] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2505.450476] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2507.565667] LustreError: 146982:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 2507.570562] LustreError: 146982:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 2507.574786] LustreError: 146982:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 7, retries 0, failed: rc = -5 [ 2507.701051] Lustre: server umount lustre-MDT0000 complete [ 2513.999711] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 2518.398982] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2536.856313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2536.984643] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2537.008223] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2537.066990] Lustre: lustre-MDT0000: new disk, initializing [ 2537.124347] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2537.137477] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2539.097393] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2545.108974] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2545.192021] Lustre: 150132:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2545.233480] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2545.236842] Lustre: Skipped 1 previous similar message [ 2545.285783] Lustre: lustre-MDT0001: new disk, initializing [ 2545.336631] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2545.361371] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2545.372226] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2547.459525] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2551.850943] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2553.903242] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2557.827079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2557.984161] Lustre: lustre-OST0000: new disk, initializing [ 2557.987227] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2557.991670] Lustre: 151869:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2558.058850] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2561.188208] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2565.469202] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2566.167286] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2566.173325] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2566.239296] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2573.148754] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 2573.961764] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 19:38:12 (1781221092) [ 2709.472935] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2709.473302] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2709.473362] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2709.479211] Lustre: Skipped 2 previous similar messages [ 2709.490597] Lustre: Skipped 2 previous similar messages [ 2711.882222] Lustre: server umount lustre-MDT0000 complete [ 2713.567508] LustreError: 150125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221231 with bad export cookie 2512843004732278868 [ 2713.568538] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2713.572638] LustreError: 150125:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2714.592769] LustreError: 150141:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2714.595356] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2714.599763] LustreError: 150141:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 2714.605910] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2718.625139] LustreError: 153508:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2718.625760] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 2719.729693] Lustre: server umount lustre-MDT0001 complete [ 2727.704653] Lustre: server umount lustre-OST0000 complete [ 2732.342055] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 2736.949494] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2755.420612] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2755.556129] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2755.570623] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2755.613574] Lustre: lustre-MDT0000: new disk, initializing [ 2755.664940] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2755.674798] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2757.305139] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2762.691659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2762.746504] Lustre: 156365:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2762.772084] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 2762.774528] Lustre: Skipped 1 previous similar message [ 2762.814687] Lustre: lustre-MDT0001: new disk, initializing [ 2762.870074] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 2762.875592] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2764.521393] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2768.607282] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2770.458353] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2773.569616] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2773.697201] Lustre: lustre-OST0000: new disk, initializing [ 2773.699772] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 2773.703531] Lustre: 158106:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2773.741135] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2773.744141] Lustre: Skipped 1 previous similar message [ 2776.340624] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2780.399145] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2781.200475] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 2781.205113] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2781.253052] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2782.606819] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 2782.727643] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 2784.965665] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 2785.097427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 2786.274872] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2786.275306] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2786.276676] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2786.279024] LustreError: Skipped 1 previous similar message [ 2786.288792] Lustre: Skipped 1 previous similar message [ 2788.207705] Lustre: server umount lustre-OST0000 complete [ 2791.394863] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2791.396697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2791.406986] Lustre: Skipped 1 previous similar message [ 2791.410338] Lustre: Skipped 1 previous similar message [ 2796.301336] Lustre: server umount lustre-MDT0000 complete [ 2796.513922] LustreError: 156373:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2796.524385] LustreError: 156373:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 2798.548755] LustreError: 157142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221316 with bad export cookie 2512843004732437684 [ 2798.551666] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2798.555852] LustreError: 157142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 2798.762753] Lustre: server umount lustre-MDT0001 complete [ 2803.486308] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:42:01 (1781221321) [ 2812.032922] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 2816.431846] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2832.589932] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2832.717453] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2832.734560] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2832.783204] Lustre: lustre-MDT0000: new disk, initializing [ 2832.825831] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2832.840843] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2834.662836] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2840.055852] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2840.121254] Lustre: 162673:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2840.268607] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2841.968105] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2845.676609] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2847.208391] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2850.237033] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2850.394272] Lustre: 164422:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2853.074318] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2856.973262] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2860.562743] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 2860.620235] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2865.634864] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2865.637074] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2865.641292] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2865.646473] Lustre: Skipped 1 previous similar message [ 2865.651560] Lustre: Skipped 1 previous similar message [ 2867.116695] Lustre: server umount lustre-OST0000 complete [ 2870.895795] Lustre: server umount lustre-MDT0000 complete [ 2872.535393] LustreError: 162663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221390 with bad export cookie 2512843004732438552 [ 2872.539561] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2872.540444] LustreError: 162663:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2872.670624] Lustre: server umount lustre-MDT0001 complete [ 2876.042621] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 19:43:14 (1781221394) [ 2882.717723] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 2887.083351] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2902.632636] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2902.793613] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 2902.824960] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 2902.828355] Lustre: Skipped 3 previous similar messages [ 2902.887309] Lustre: lustre-MDT0000: new disk, initializing [ 2902.890108] Lustre: Skipped 2 previous similar messages [ 2902.940171] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2902.942640] Lustre: Skipped 2 previous similar messages [ 2902.955423] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 2902.962173] Lustre: Skipped 2 previous similar messages [ 2905.061398] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2910.748320] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2910.813551] Lustre: 168586:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 2910.960213] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 2912.841087] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2916.781962] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2918.579787] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2921.529269] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2921.669194] Lustre: 170336:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 2924.233152] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2925.818654] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 2927.816080] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2931.173476] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2931.175644] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2931.175754] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2931.175768] Lustre: Skipped 3 previous similar messages [ 2931.178517] Lustre: Skipped 3 previous similar messages [ 2931.181079] LustreError: Skipped 1 previous similar message [ 2936.554513] Lustre: server umount lustre-OST0000 complete [ 2944.629543] Lustre: server umount lustre-MDT0000 complete [ 2946.260362] LustreError: 168580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221464 with bad export cookie 2512843004732439623 [ 2946.263257] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2946.264927] LustreError: 168580:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2946.405679] Lustre: server umount lustre-MDT0001 complete [ 2950.787746] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 2952.195641] Key type lgssc unregistered [ 2952.379607] LNet: 172059:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2952.383841] LNetError: 172059:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2953.449065] LNet: Removed LNI 192.168.206.140@tcp [ 2953.855256] Key type .llcrypt unregistered [ 2953.856957] Key type ._llcrypt unregistered [ 2964.726791] Key type ._llcrypt registered [ 2964.728375] Key type .llcrypt registered [ 2964.785137] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 2965.261175] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2965.318922] alg: No test for adler32 (adler32-zlib) [ 2966.236900] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 2966.344739] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 2967.944193] Key type lgssc registered [ 2968.513321] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2972.882468] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2972.895782] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2974.169233] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2976.138069] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2980.349642] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2980.578516] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2982.655787] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2985.404256] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 2986.984686] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 2989.860361] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2990.041960] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2992.526814] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 2994.822479] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 2999.273789] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3014.635676] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3014.636724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3014.643514] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3014.649829] Lustre: Skipped 1 previous similar message [ 3014.663301] Lustre: Skipped 1 previous similar message [ 3019.749663] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3019.752159] Lustre: Skipped 1 previous similar message [ 3019.814397] Lustre: server umount lustre-OST0000 complete [ 3021.793346] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3021.799762] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3021.810084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3024.865750] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3024.866996] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3027.841586] Lustre: server umount lustre-MDT0000 complete [ 3029.903021] LustreError: 173415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221548 with bad export cookie 17455396326704890790 [ 3029.906210] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3029.907920] LustreError: 173415:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3030.126578] Lustre: server umount lustre-MDT0001 complete [ 3036.070634] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 3037.581676] Key type lgssc unregistered [ 3037.756716] LNet: 176629:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3037.761383] LNetError: 176629:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3037.778104] LNet: Removed LNI 192.168.206.140@tcp [ 3038.303723] Key type .llcrypt unregistered [ 3038.305403] Key type ._llcrypt unregistered [ 3050.944848] Key type ._llcrypt registered [ 3050.946911] Key type .llcrypt registered [ 3051.007562] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3051.428726] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3051.438631] alg: No test for adler32 (adler32-zlib) [ 3052.355473] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 3052.491967] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 3054.080215] Key type lgssc registered [ 3054.594755] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3058.528253] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3058.545190] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3059.864953] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3062.114121] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3066.150960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3066.398708] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3068.565580] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3071.960677] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3074.028625] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3077.466135] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3077.639056] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3080.589802] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3083.133143] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3084.780943] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 3100.017507] Lustre: lustre-OST0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 3100.131677] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3100.139184] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.153718] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3101.156913] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3101.161792] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3101.166404] Lustre: Skipped 1 previous similar message [ 3105.119900] Lustre: lustre-OST0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 3105.124109] Lustre: Skipped 1 previous similar message [ 3110.243031] Lustre: lustre-OST0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 3110.247145] Lustre: Skipped 2 previous similar messages [ 3110.368948] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3110.460466] Lustre: server umount lustre-OST0000 complete [ 3113.142318] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8e6e06ac1800.ost_server_uuid 50 [ 3113.883564] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-ffff8e6e06ac1800.ost_server_uuid in DISCONN state after 0 sec [ 3114.702728] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 3126.757038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3126.761267] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3126.768053] Lustre: Skipped 1 previous similar message [ 3126.780130] Lustre: Skipped 1 previous similar message [ 3132.030673] Lustre: server umount lustre-MDT0000 complete [ 3133.955965] LustreError: 177986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221652 with bad export cookie 1879610549772730286 [ 3133.960605] LustreError: 177986:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3133.961576] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3134.131227] Lustre: server umount lustre-MDT0001 complete [ 3138.134846] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 19:47:36 (1781221656) [ 3141.486393] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3141.718823] LustreError: 181543:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3141.730448] LustreError: 181543:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3141.763773] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3143.462176] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3147.235522] LustreError: 181544:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3147.786235] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3147.975082] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3150.157720] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3153.063232] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3154.841287] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3158.495427] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3158.698274] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3161.562141] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3164.440415] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3166.892803] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 3167.693510] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3167.784022] Lustre: lustre-OST0001: new disk, initializing [ 3167.786129] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3167.790551] Lustre: 184041:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3170.402618] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3170.407715] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3170.444040] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3170.542442] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3174.593854] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3177.442933] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3177.450656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3177.457283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3177.460305] Lustre: Skipped 2 previous similar messages [ 3190.240229] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3190.277097] Lustre: server umount lustre-OST0000 complete [ 3192.746149] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3192.846929] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3194.338847] LustreError: 183137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3194.344961] LustreError: 183137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3194.558437] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3194.645716] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3199.465827] LustreError: 183135:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3199.481017] LustreError: 183135:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3204.582606] LustreError: 183137:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3204.592779] LustreError: 183137:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3206.631958] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3206.632450] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3206.635651] LustreError: Skipped 2 previous similar messages [ 3206.637148] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3206.647231] Lustre: Skipped 2 previous similar messages [ 3206.649201] Lustre: Skipped 9 previous similar messages [ 3211.567718] Lustre: server umount lustre-OST0001 complete [ 3214.820173] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3214.820954] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3214.831689] Lustre: Skipped 1 previous similar message [ 3220.122547] Lustre: server umount lustre-MDT0000 complete [ 3222.269895] LustreError: 181524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221740 with bad export cookie 1879610549772731357 [ 3222.272028] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3222.281124] LustreError: 181524:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3222.482373] Lustre: server umount lustre-MDT0001 complete [ 3235.624381] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 19:49:13 (1781221753) [ 3239.120037] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3239.267525] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3239.284982] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3239.374204] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3239.377234] Lustre: Skipped 1 previous similar message [ 3241.750718] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3246.343858] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3246.422197] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3246.446905] Lustre: 187312:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3248.734157] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3251.706668] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3253.476122] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3257.069075] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3257.201399] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3260.075546] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3262.504093] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3263.004108] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3265.682086] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3265.726062] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3268.582961] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3271.170047] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3273.186830] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3273.187079] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3273.190547] LustreError: Skipped 1 previous similar message [ 3273.198721] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3273.205551] Lustre: Skipped 8 previous similar messages [ 3278.486343] Lustre: server umount lustre-OST0000 complete [ 3281.760812] LustreError: 189526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3281.772173] LustreError: 189526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3293.666976] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3293.672446] LustreError: Skipped 1 previous similar message [ 3297.004414] Lustre: server umount lustre-OST0001 complete [ 3313.122424] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3313.238503] Lustre: server umount lustre-MDT0000 complete [ 3315.248605] LustreError: 187303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221833 with bad export cookie 1879610549772732750 [ 3315.249732] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3315.259788] LustreError: 187303:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 3315.470884] Lustre: server umount lustre-MDT0001 complete [ 3326.310831] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 19:50:44 (1781221844) [ 3333.607490] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 3337.636977] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3354.350177] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3354.491523] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3354.513308] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3354.556286] Lustre: lustre-MDT0000: new disk, initializing [ 3354.602493] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3354.604930] Lustre: Skipped 3 previous similar messages [ 3354.619556] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3357.162714] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3365.089531] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3365.215442] Lustre: 194905:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3365.263517] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3365.266886] Lustre: Skipped 1 previous similar message [ 3365.324671] Lustre: lustre-MDT0001: new disk, initializing [ 3365.436082] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3365.447978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3368.562981] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3374.418492] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3377.095784] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3381.086265] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3381.254643] Lustre: lustre-OST0000: new disk, initializing [ 3381.259994] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3381.266575] Lustre: 196646:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3384.616816] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3389.081469] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3389.500781] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3389.509104] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3389.572680] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3391.430434] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3391.541854] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3393.625564] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3393.743626] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3395.042561] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3395.043162] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3395.046712] LustreError: Skipped 1 previous similar message [ 3395.048566] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3395.056832] Lustre: Skipped 6 previous similar messages [ 3395.066978] Lustre: Skipped 14 previous similar messages [ 3409.376741] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3409.514433] Lustre: server umount lustre-OST0000 complete [ 3417.506773] Lustre: server umount lustre-MDT0000 complete [ 3419.383192] LustreError: 194896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781221937 with bad export cookie 1879610549772734269 [ 3419.387241] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3419.391674] LustreError: 194896:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3419.528588] Lustre: server umount lustre-MDT0001 complete [ 3423.416307] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3423.652420] LustreError: 198713:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3423.670187] LustreError: 198713:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 7 previous similar messages [ 3423.695242] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3423.697806] Lustre: Skipped 2 previous similar messages [ 3425.635561] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3430.335500] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3432.970633] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3435.968674] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3437.821117] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3441.542081] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3444.693993] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3447.321210] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3449.266960] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3449.370757] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3451.201628] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3455.459909] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 4 sec [ 3459.044738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3459.050531] LustreError: Skipped 1 previous similar message [ 3459.052608] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3459.059784] Lustre: Skipped 2 previous similar messages [ 3462.562099] Lustre: server umount lustre-OST0000 complete [ 3465.331334] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3467.282687] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3492.283174] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3495.685061] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3498.223208] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3500.710316] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3500.716651] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3509.928492] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3509.936308] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3514.666948] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 15 sec [ 3522.297587] Lustre: server umount lustre-OST0000 complete [ 3525.603334] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3525.606280] Lustre: Skipped 14 previous similar messages [ 3538.400970] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3538.481299] Lustre: server umount lustre-MDT0000 complete [ 3540.753440] LustreError: 201476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222058 with bad export cookie 1879610549772735151 [ 3540.757440] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3540.762745] LustreError: 201476:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3540.963349] Lustre: server umount lustre-MDT0001 complete [ 3546.593345] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 19:54:24 (1781222064) [ 3550.616053] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3550.857844] LustreError: 204209:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3550.865161] LustreError: 204209:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 3553.084220] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3558.487051] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3558.717531] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3558.724989] Lustre: Skipped 4 previous similar messages [ 3560.746867] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3563.580682] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3565.542660] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3569.469350] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3573.555305] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3576.919883] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3579.240448] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3579.377424] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3581.641248] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3583.822528] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 2 sec [ 3587.472675] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3587.571372] Lustre: lustre-OST0001: new disk, initializing [ 3587.574427] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3587.580056] Lustre: 207220:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3590.566357] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3593.747499] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3593.752322] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3593.805873] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3594.875955] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3597.319223] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3597.444485] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3599.670546] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3599.769637] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3603.943651] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3603.948285] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3603.951654] Lustre: Skipped 6 previous similar messages [ 3603.955239] LustreError: Skipped 3 previous similar messages [ 3615.200349] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3615.254387] Lustre: server umount lustre-OST0001 complete [ 3617.982287] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3618.090904] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 3619.301955] LustreError: 205803:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3619.312834] LustreError: 205803:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3620.248836] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3620.367746] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 3646.132274] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3649.171107] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3651.559789] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3651.567982] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 3651.568171] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3651.584566] Lustre: Skipped 1 previous similar message [ 3652.793867] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3654.223936] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3661.526397] Lustre: server umount lustre-OST0001 complete [ 3681.649688] Lustre: server umount lustre-OST0000 complete [ 3689.717458] Lustre: server umount lustre-MDT0000 complete [ 3691.462524] LustreError: 204235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222209 with bad export cookie 1879610549772736103 [ 3691.467043] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3691.467886] LustreError: 204235:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 3697.683111] Lustre: server umount lustre-MDT0001 complete [ 3709.368649] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 19:57:07 (1781222227) [ 3712.505507] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3712.655687] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3712.671156] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3714.846927] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3718.735747] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3718.798696] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3718.817327] Lustre: 212160:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3720.715683] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3723.110462] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3724.778462] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3727.615858] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3727.699989] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 3730.641916] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3733.164304] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3738.147603] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3738.211466] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 3741.297836] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3744.457980] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3745.344828] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 3747.789666] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3747.911458] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3750.459569] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3750.570741] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3752.374549] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8e6e04602800.ost_server_uuid 50 [ 3753.063958] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8e6e04602800.ost_server_uuid in FULL state after 0 sec [ 3753.571528] Lustre: Permanently deactivating lustre-OST0001 [ 3753.584098] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 3755.206667] Lustre: Permanently reactivating lustre-OST0001 [ 3763.345603] Lustre: server umount lustre-OST0001 complete [ 3765.220485] LustreError: 215219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3765.228377] LustreError: 215219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 25 previous similar messages [ 3771.302625] Lustre: server umount lustre-OST0000 complete [ 3785.697301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3785.700774] Lustre: Skipped 30 previous similar messages [ 3787.232151] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3787.321594] Lustre: server umount lustre-MDT0000 complete [ 3789.232788] LustreError: 214692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222307 with bad export cookie 1879610549772737391 [ 3789.235747] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3789.241504] LustreError: 214692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3789.390474] Lustre: server umount lustre-MDT0001 complete [ 3803.822750] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 19:58:41 (1781222321) [ 3807.958032] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3808.109150] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 3808.124801] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3808.132860] Lustre: Skipped 5 previous similar messages [ 3810.554702] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3814.481805] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3814.542935] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 3814.555133] Lustre: 218267:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3816.401675] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3818.861096] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3820.623312] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3823.898515] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3824.037614] Lustre: Permanently deactivating lustre-OST0000 [ 3824.045918] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 3824.121516] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3824.125164] Lustre: Skipped 9 previous similar messages [ 3825.918735] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 3826.825104] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3829.240535] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3832.106948] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3834.532587] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 3834.657698] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3837.012589] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3837.938518] Lustre: Permanently reactivating lustre-OST0000 [ 3843.079669] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 3843.102720] Lustre: lustre-OST0000: Client lustre-MDT0000-mdtlov_UUID (at 0@lo) reconnecting [ 3843.106741] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 3843.117433] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 3843.121944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 3853.461410] Lustre: server umount lustre-OST0001 complete [ 3870.177943] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3870.257435] Lustre: server umount lustre-OST0000 complete [ 3875.810713] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3875.817413] Lustre: Skipped 17 previous similar messages [ 3878.555247] Lustre: server umount lustre-MDT0000 complete [ 3880.688302] LustreError: 218259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222398 with bad export cookie 1879610549772739015 [ 3880.691464] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3880.695069] LustreError: 218259:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3880.860683] Lustre: server umount lustre-MDT0001 complete [ 3885.716117] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 3887.144757] Key type lgssc unregistered [ 3887.311412] LNet: 222647:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3887.323188] LNetError: 222647:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3887.338607] LNet: Removed LNI 192.168.206.140@tcp [ 3887.773254] Key type .llcrypt unregistered [ 3887.775329] Key type ._llcrypt unregistered [ 3898.283159] Key type ._llcrypt registered [ 3898.286090] Key type .llcrypt registered [ 3898.339546] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 3898.866740] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3898.885929] alg: No test for adler32 (adler32-zlib) [ 3899.764996] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 3899.869549] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 3901.456189] Key type lgssc registered [ 3901.961918] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3906.677209] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3906.686737] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3907.926297] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3909.607393] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3913.302792] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3913.369929] Lustre: Found index 1 for lustre-MDT0001, updating log [ 3913.391951] Lustre: 224064:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3913.398159] Lustre: Permanently deactivating lustre-MDT0001 [ 3913.407959] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3913.560659] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3915.216932] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3916.914893] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3916.922389] LustreError: 224105:0:(osp_object.c:617:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3916.929543] LustreError: 224105:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3916.935568] LustreError: 224105:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 9, retries 1, failed: rc = -108 [ 3917.806920] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 3919.790373] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3923.339825] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3923.526154] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3924.580872] LustreError: 225687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3924.586624] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 3924.588389] LustreError: 225687:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3925.989679] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3928.219303] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 3930.082950] LustreError: 225687:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3931.126318] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3931.240431] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3933.290238] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 3933.292885] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 3933.296075] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 3933.712464] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 3936.089601] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 3938.965698] Lustre: Permanently reactivating lustre-MDT0001 [ 3938.974673] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 3938.978930] Lustre: Skipped 2 previous similar messages [ 3943.795279] Lustre: lustre-MDT0001: Client 733a717e-a2cd-4bc4-8306-9bce1fb86b64 (at 192.168.206.40@tcp) reconnecting [ 3946.101382] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 3946.881339] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 3949.223152] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 3949.336428] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 3951.324603] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 3951.421376] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 3952.652311] Lustre: Permanently deactivating lustre-MDT0001 [ 3952.662645] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 3952.665978] Lustre: Skipped 2 previous similar messages [ 3956.502376] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 3963.880805] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 3963.885966] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3963.893705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3964.018709] Lustre: server umount lustre-MDT0000 complete [ 3964.387549] LustreError: 224074:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3964.390426] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3964.395788] LustreError: 224074:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 3964.405307] Lustre: Skipped 2 previous similar messages [ 3965.723580] LustreError: 224056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222483 with bad export cookie 10792654283170841313 [ 3965.726058] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3965.731805] LustreError: 224056:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3965.883887] Lustre: server umount lustre-MDT0001 complete [ 3967.589081] Lustre: server umount lustre-OST0000 complete [ 3973.120829] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 20:01:31 (1781222491) [ 3978.425995] Lustre: server umount lustre-OST0001 complete [ 3981.350320] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 3986.122921] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4002.516825] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4002.641765] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4002.648224] Lustre: Skipped 2 previous similar messages [ 4002.665946] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4002.710425] Lustre: lustre-MDT0000: new disk, initializing [ 4002.759891] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4002.772351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4004.928264] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4011.361319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4011.428525] Lustre: 231762:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4011.471258] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4011.474486] Lustre: Skipped 1 previous similar message [ 4011.520523] Lustre: lustre-MDT0001: new disk, initializing [ 4011.576440] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4011.608818] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4011.617273] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4013.439808] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4017.682929] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4019.524677] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4023.052204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4023.200421] Lustre: lustre-OST0000: new disk, initializing [ 4023.203956] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4023.208911] Lustre: 233502:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4025.908658] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4026.190777] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4026.196329] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4026.259323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4029.836589] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4030.804919] LustreError: 231769:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4036.033065] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4036.132429] Lustre: lustre-OST0001: new disk, initializing [ 4036.134784] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4036.139313] Lustre: 234504:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4036.184651] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4036.191087] Lustre: Skipped 1 previous similar message [ 4039.250257] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4040.816143] LustreError: 231769:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4040.831344] LustreError: 234158:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 4041.756709] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4041.765981] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4041.809482] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4043.732643] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 4044.808665] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 4050.928127] LustreError: 234158:0:(mdt_handler.c:3266:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 4053.883984] Lustre: lustre-OST0001: Not available for connect from 192.168.206.40@tcp (stopping) [ 4057.062906] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4057.063891] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4057.067036] LustreError: Skipped 1 previous similar message [ 4057.072894] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4057.077029] Lustre: Skipped 1 previous similar message [ 4058.260736] Lustre: server umount lustre-OST0001 complete [ 4058.979378] LustreError: 233484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4058.989617] LustreError: 233484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4067.303346] LustreError: 233484:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4067.312800] LustreError: 233484:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4072.420020] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4072.428789] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4072.438345] Lustre: Skipped 1 previous similar message [ 4072.442975] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4072.446515] Lustre: Skipped 1 previous similar message [ 4077.546404] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4077.547241] LustreError: 233872:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4077.554205] Lustre: Skipped 1 previous similar message [ 4077.563763] LustreError: 233872:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4078.219831] Lustre: server umount lustre-OST0000 complete [ 4082.658895] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4082.671607] Lustre: Skipped 2 previous similar messages [ 4086.823028] Lustre: server umount lustre-MDT0000 complete [ 4088.907678] LustreError: 231753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222607 with bad export cookie 10792654283170844113 [ 4088.910390] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4088.914388] LustreError: 231753:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4095.136596] Lustre: server umount lustre-MDT0001 complete [ 4100.373896] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 4101.856942] Key type lgssc unregistered [ 4102.016868] LNet: 236403:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4102.022163] LNetError: 236403:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4102.032474] LNet: Removed LNI 192.168.206.140@tcp [ 4102.445670] Key type .llcrypt unregistered [ 4102.448165] Key type ._llcrypt unregistered [ 4121.824134] Key type ._llcrypt registered [ 4121.826129] Key type .llcrypt registered [ 4121.896969] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4122.444531] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4122.465436] alg: No test for adler32 (adler32-zlib) [ 4123.377153] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 4123.499271] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 4125.112335] Key type lgssc registered [ 4125.642234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4129.387409] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4129.398865] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4130.552867] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4130.566858] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4130.655256] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4132.677399] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4137.078290] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4137.139229] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4137.161689] Lustre: 238568:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4137.317187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4139.455458] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4142.273232] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4144.222802] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4147.581483] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4147.679789] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4147.800767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4150.381764] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 4150.800519] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4153.448514] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4196.321218] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4196.331363] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.347714] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4196.834375] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4196.836869] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4196.852346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4200.372287] Lustre: server umount lustre-OST0000 complete [ 4202.571703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 4208.487138] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4208.669458] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4211.475043] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4214.384572] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4217.894347] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4217.901853] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 4217.901943] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 4253.670364] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4253.672085] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4253.672467] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4253.681407] Lustre: Skipped 1 previous similar message [ 4255.294364] Lustre: server umount lustre-OST0000 complete [ 4258.788917] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4258.790725] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4258.797168] Lustre: Skipped 1 previous similar message [ 4258.799665] Lustre: Skipped 1 previous similar message [ 4263.341461] Lustre: server umount lustre-MDT0000 complete [ 4263.906542] LustreError: 238582:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4263.915755] LustreError: 238582:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4265.208129] LustreError: 238563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222783 with bad export cookie 3757685587013337440 [ 4265.211457] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4265.214057] LustreError: 238563:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4265.385624] Lustre: server umount lustre-MDT0001 complete [ 4270.237266] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 4271.557036] Key type lgssc unregistered [ 4271.708501] LNet: 243518:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4271.713322] LNetError: 243518:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4271.725577] LNet: Removed LNI 192.168.206.140@tcp [ 4272.122354] Key type .llcrypt unregistered [ 4272.123514] Key type ._llcrypt unregistered [ 4284.056842] Key type ._llcrypt registered [ 4284.058169] Key type .llcrypt registered [ 4284.112987] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4284.701234] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4284.712745] alg: No test for adler32 (adler32-zlib) [ 4285.603160] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 4285.711093] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 4287.320202] Key type lgssc registered [ 4287.859070] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4291.690267] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4291.699181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4292.951984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4294.880462] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4299.174223] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4299.365152] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4301.167524] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4303.834943] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4305.414678] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4308.305946] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4308.472479] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4311.018190] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4313.416760] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4318.700697] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 4329.954054] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4329.957940] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4329.964758] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4334.051132] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4334.051895] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4334.053443] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4335.071837] Lustre: server umount lustre-OST0000 complete [ 4339.169950] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4339.170744] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4339.176616] Lustre: Skipped 1 previous similar message [ 4339.179280] Lustre: Skipped 1 previous similar message [ 4342.769140] Lustre: server umount lustre-MDT0000 complete [ 4344.290407] LustreError: 244894:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4344.298482] LustreError: 244894:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4344.418374] LustreError: 244874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222862 with bad export cookie 15124477245538542129 [ 4344.421520] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4344.424932] LustreError: 244874:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4344.560065] Lustre: server umount lustre-MDT0001 complete [ 4348.830948] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 4350.083228] Key type lgssc unregistered [ 4350.230682] LNet: 248838:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4350.234897] LNetError: 248838:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4350.244647] LNet: Removed LNI 192.168.206.140@tcp [ 4350.586964] Key type .llcrypt unregistered [ 4350.588700] Key type ._llcrypt unregistered [ 4358.673277] Key type ._llcrypt registered [ 4358.674765] Key type .llcrypt registered [ 4358.727202] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4359.125594] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4359.177626] alg: No test for adler32 (adler32-zlib) [ 4360.056346] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 4360.159114] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 4361.760319] Key type lgssc registered [ 4362.232810] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4365.629928] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4365.640658] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4366.860517] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4368.643706] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4372.973225] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4373.186139] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4375.174258] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4377.947397] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4379.928973] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4383.354778] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4383.460925] LustreError: 251610:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4383.469979] LustreError: 251610:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 4383.543501] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4386.591522] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4389.607924] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4390.891335] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4401.121312] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4401.126344] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4401.134347] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4404.194199] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4404.195319] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4404.199365] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4406.699949] Lustre: server umount lustre-OST0000 complete [ 4409.313073] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4409.317310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4409.318890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4409.327976] Lustre: Skipped 1 previous similar message [ 4409.336581] Lustre: Skipped 2 previous similar messages [ 4414.434956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4414.439751] Lustre: Skipped 1 previous similar message [ 4419.553480] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4419.559143] Lustre: Skipped 1 previous similar message [ 4423.136275] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4423.219153] Lustre: server umount lustre-MDT0000 complete [ 4424.673250] LustreError: 250021:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4424.684124] LustreError: 250021:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 4425.039234] LustreError: 250001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781222943 with bad export cookie 5957909113082433336 [ 4425.042508] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4425.046535] LustreError: 250001:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4425.201515] Lustre: server umount lustre-MDT0001 complete [ 4429.923315] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 4431.153254] Key type lgssc unregistered [ 4431.305171] LNet: 253343:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4431.308841] LNetError: 253343:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4431.323258] LNet: Removed LNI 192.168.206.140@tcp [ 4431.694622] Key type .llcrypt unregistered [ 4431.696291] Key type ._llcrypt unregistered [ 4443.500229] Key type ._llcrypt registered [ 4443.502188] Key type .llcrypt registered [ 4443.554678] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4444.036869] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4444.051735] alg: No test for adler32 (adler32-zlib) [ 4444.949080] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 4445.057963] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 4446.672154] Key type lgssc registered [ 4447.217253] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4450.856185] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4450.868029] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4452.078362] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4453.741153] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4457.688247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4457.885151] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4459.679448] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4462.274211] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4464.212083] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4467.409039] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4467.592477] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4470.274690] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4472.974261] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4477.868510] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4497.382245] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4497.386751] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4497.388730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4497.397747] Lustre: Skipped 1 previous similar message [ 4500.186415] Lustre: server umount lustre-OST0000 complete [ 4502.498670] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4502.499985] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4502.506098] Lustre: Skipped 1 previous similar message [ 4502.510864] Lustre: Skipped 2 previous similar messages [ 4507.617092] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4507.621891] Lustre: Skipped 1 previous similar message [ 4512.737442] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4512.741379] Lustre: Skipped 1 previous similar message [ 4516.320160] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4516.394962] Lustre: server umount lustre-MDT0000 complete [ 4517.857400] LustreError: 255837:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4517.870250] LustreError: 255837:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4517.886312] LustreError: 254700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223036 with bad export cookie 16461524472904990775 [ 4517.889490] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4517.893282] LustreError: 254700:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4518.047444] Lustre: server umount lustre-MDT0001 complete [ 4523.552823] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 4525.270835] Key type lgssc unregistered [ 4525.450061] LNet: 258688:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4525.455342] LNetError: 258688:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4525.464475] LNet: Removed LNI 192.168.206.140@tcp [ 4525.949778] Key type .llcrypt unregistered [ 4525.952163] Key type ._llcrypt unregistered [ 4534.919227] Key type ._llcrypt registered [ 4534.921117] Key type .llcrypt registered [ 4534.984776] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4535.491585] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4535.531808] alg: No test for adler32 (adler32-zlib) [ 4536.432541] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 4536.546656] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 4538.152220] Key type lgssc registered [ 4538.761835] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4542.473869] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4542.489771] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4543.739059] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4545.302150] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4549.011399] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4549.181773] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4550.841566] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4553.555859] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4555.559281] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4559.602226] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4559.807444] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4563.082294] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4565.724376] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4568.046318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 4582.883344] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 4582.888905] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4582.897206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4584.419207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4584.421817] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4584.432625] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4586.182806] Lustre: server umount lustre-OST0000 complete [ 4589.538918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4589.553947] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4589.559107] Lustre: Skipped 2 previous similar messages [ 4594.627599] Lustre: server umount lustre-MDT0000 complete [ 4594.658056] LustreError: 259885:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4594.668979] LustreError: 259885:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4596.404405] LustreError: 260626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223114 with bad export cookie 10269656011933005275 [ 4596.409488] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4596.410078] LustreError: 260626:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4596.559612] Lustre: server umount lustre-MDT0001 complete [ 4601.228421] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 4602.363463] Key type lgssc unregistered [ 4602.504489] LNet: 263246:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4602.507618] LNetError: 263246:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4602.516410] LNet: Removed LNI 192.168.206.140@tcp [ 4602.840208] Key type .llcrypt unregistered [ 4602.841865] Key type ._llcrypt unregistered [ 4606.795761] Key type ._llcrypt registered [ 4606.797073] Key type .llcrypt registered [ 4606.846989] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 4613.046778] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 4621.407223] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4621.977088] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4622.009565] alg: No test for adler32 (adler32-zlib) [ 4622.942512] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 4623.066923] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 4624.680123] Key type lgssc registered [ 4625.378568] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4639.567141] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4639.573974] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4640.692215] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4640.715745] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4640.756681] Lustre: lustre-MDT0000: new disk, initializing [ 4640.802771] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4640.815932] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4642.432961] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4648.471740] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4648.548520] Lustre: 266721:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4648.593458] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4648.597789] Lustre: Skipped 1 previous similar message [ 4648.661462] Lustre: lustre-MDT0001: new disk, initializing [ 4648.707732] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4648.732197] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4648.738327] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4650.911294] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4655.628105] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4657.505524] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4660.751850] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4660.891894] Lustre: lustre-OST0000: new disk, initializing [ 4660.894760] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4660.898970] Lustre: 268464:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4660.940162] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4663.287405] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4666.621705] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4668.191411] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4669.449309] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4669.453708] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4669.499323] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4670.368952] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 2 sec [ 4672.039040] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4672.139653] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4674.530410] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4674.531084] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4674.535605] LustreError: Skipped 1 previous similar message [ 4674.544861] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4674.548388] Lustre: Skipped 1 previous similar message [ 4679.071116] Lustre: server umount lustre-OST0000 complete [ 4684.772676] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4684.772896] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4684.779890] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4684.789900] Lustre: Skipped 2 previous similar messages [ 4684.795774] Lustre: Skipped 1 previous similar message [ 4687.581728] Lustre: server umount lustre-MDT0000 complete [ 4689.993996] LustreError: 268463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223208 with bad export cookie 12491112295898583365 [ 4689.997328] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4690.004974] LustreError: 268463:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4690.019777] LustreError: 264948:0:(import.c:719:ptlrpc_connect_import_locked()) already connecting [ 4690.231761] Lustre: server umount lustre-MDT0001 complete [ 4694.642736] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 20:13:32 (1781223212) [ 4697.675301] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4697.914157] LustreError: 270739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4697.922914] LustreError: 270739:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 4697.958824] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4699.826137] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4703.202597] LustreError: 270739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4703.659149] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4703.849804] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4705.491684] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4708.037794] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4710.077992] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4713.767596] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4713.975420] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4717.458613] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4720.218433] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4738.530318] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4738.534087] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4738.543091] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4738.561905] Lustre: Skipped 1 previous similar message [ 4740.810933] Lustre: server umount lustre-OST0000 complete [ 4743.653404] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4743.655447] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4743.664570] Lustre: Skipped 1 previous similar message [ 4743.667301] Lustre: Skipped 1 previous similar message [ 4745.891625] Lustre: server umount lustre-MDT0000 complete [ 4748.301159] LustreError: 270720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223266 with bad export cookie 12491112295898584233 [ 4748.308368] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4748.308465] LustreError: 270720:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 4748.528777] Lustre: server umount lustre-MDT0001 complete [ 4754.717322] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 4756.335654] Key type lgssc unregistered [ 4756.503883] LNet: 273933:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4756.506657] LNetError: 273933:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4756.515260] LNet: Removed LNI 192.168.206.140@tcp [ 4756.986906] Key type .llcrypt unregistered [ 4756.989784] Key type ._llcrypt unregistered [ 4769.906973] Key type ._llcrypt registered [ 4769.909577] Key type .llcrypt registered [ 4769.984324] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 4778.102990] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4778.642130] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4778.732748] alg: No test for adler32 (adler32-zlib) [ 4779.674455] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 4779.829565] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 4781.440306] Key type lgssc registered [ 4782.088966] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4803.302849] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4803.312731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4804.455224] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4804.475607] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4804.518784] Lustre: lustre-MDT0000: new disk, initializing [ 4804.562665] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4804.573157] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4806.421613] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4812.353451] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4812.433799] Lustre: 277699:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4812.481627] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4812.485025] Lustre: Skipped 1 previous similar message [ 4812.540725] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 4812.549233] Lustre: lustre-MDT0001: new disk, initializing [ 4812.612106] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4812.640391] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4812.649547] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4814.732477] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4819.033539] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4820.952605] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4824.385827] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4824.545721] Lustre: lustre-OST2710: new disk, initializing [ 4824.549151] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 4824.553896] Lustre: 279441:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4824.610158] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 4827.586852] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4829.714701] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 4829.719767] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 4829.776744] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 4831.907168] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid [ 4835.613473] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4835.740172] Lustre: lustre-OST03e8: new disk, initializing [ 4835.742529] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 4835.745873] Lustre: 280391:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4835.814263] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 4836.851049] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 4836.859738] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 4836.914536] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 4839.130165] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4842.398918] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid [ 4847.219755] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 4847.330410] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4849.471426] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 4849.599278] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4851.768864] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 4851.880694] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4854.159929] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 4854.282879] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4857.314740] Lustre: lustre-MDT0000-lwp-OST2710: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4857.317371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4857.324906] Lustre: Skipped 3 previous similar messages [ 4857.331858] Lustre: Skipped 3 previous similar messages [ 4862.385588] Lustre: server umount lustre-MDT0000 complete [ 4864.257863] LustreError: 277692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223382 with bad export cookie 15880269718316322272 [ 4864.260375] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4864.264327] LustreError: 277692:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4864.461239] Lustre: server umount lustre-MDT0001 complete [ 4876.826734] Lustre: server umount lustre-OST2710 complete [ 4888.963373] Lustre: server umount lustre-OST03e8 complete [ 4897.604709] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 4902.306393] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4921.333068] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4921.476026] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4921.498840] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4921.548879] Lustre: lustre-MDT0000: new disk, initializing [ 4921.595500] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4921.607667] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4923.763275] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4929.421557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4929.474639] Lustre: 285420:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4929.506497] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4929.509648] Lustre: Skipped 1 previous similar message [ 4929.551618] Lustre: lustre-MDT0001: new disk, initializing [ 4929.611147] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4929.632933] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4929.638949] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4931.717564] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4936.058604] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 4937.800836] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 4941.119829] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4941.261733] Lustre: 287161:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4942.157226] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4942.237509] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4944.083494] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4948.268221] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 4950.644398] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4950.778925] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4952.827203] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4952.958719] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4957.667558] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4957.669559] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4957.673888] LustreError: Skipped 1 previous similar message [ 4957.676141] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4957.679388] Lustre: Skipped 1 previous similar message [ 4960.097753] Lustre: server umount lustre-OST0000 complete [ 4962.786356] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4962.796978] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4962.799889] Lustre: Skipped 2 previous similar messages [ 4967.906286] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4967.912174] Lustre: Skipped 1 previous similar message [ 4968.186924] Lustre: server umount lustre-MDT0000 complete [ 4970.107186] LustreError: 285412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223488 with bad export cookie 15880269718316323952 [ 4970.108019] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4970.115621] LustreError: 285412:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4970.266709] Lustre: server umount lustre-MDT0001 complete [ 4974.290325] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 20:18:12 (1781223492) [ 4975.022282] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 4975.841164] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 20:18:14 (1781223494) [ 4977.297491] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 4989.436455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4989.602121] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4989.614113] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4989.696453] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4989.698907] Lustre: Skipped 1 previous similar message [ 4991.448867] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 4995.264490] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4995.348313] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4995.364654] Lustre: 290846:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4997.466485] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5014.440442] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5014.588230] Lustre: Denying initial registration attempt from nid 192.168.206.140@tcp, specified as failover [ 5014.595484] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 5014.604691] LustreError: 292266:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -99 [ 5014.607912] LustreError: 292266:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 5014.611818] LustreError: 292266:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5014.650173] Lustre: server umount lustre-OST0000 complete [ 5014.652835] LustreError: 292266:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 5021.153851] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5021.157967] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5021.164641] Lustre: Skipped 1 previous similar message [ 5021.167959] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5024.276498] Lustre: server umount lustre-MDT0000 complete [ 5026.257723] LustreError: 290841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223544 with bad export cookie 15880269718316324834 [ 5026.258827] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5026.267398] LustreError: 290841:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5026.456272] Lustre: server umount lustre-MDT0001 complete [ 5031.347140] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5032.796529] Key type lgssc unregistered [ 5032.963800] LNet: 293369:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5032.968931] LNetError: 293369:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5032.976721] LNet: Removed LNI 192.168.206.140@tcp [ 5033.412590] Key type .llcrypt unregistered [ 5033.415156] Key type ._llcrypt unregistered [ 5038.689359] Key type ._llcrypt registered [ 5038.691189] Key type .llcrypt registered [ 5038.745715] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5039.331810] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5039.348162] alg: No test for adler32 (adler32-zlib) [ 5040.280421] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 5040.424521] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5042.024231] Key type lgssc registered [ 5042.568577] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5070.859578] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5075.148785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5075.161919] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5076.332293] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5076.350138] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5076.446985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5078.521058] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5083.231213] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5083.304605] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5083.322866] Lustre: 296072:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5083.483059] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5085.626569] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5088.478315] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5090.321550] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5106.728321] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5106.860403] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5106.966127] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5109.774454] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5112.159342] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5117.923855] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5117.923885] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5117.936346] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5119.224604] Lustre: server umount lustre-OST0000 complete [ 5123.042029] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5123.043426] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5123.048426] Lustre: Skipped 2 previous similar messages [ 5123.053447] Lustre: Skipped 2 previous similar messages [ 5128.162127] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5128.165358] Lustre: Skipped 1 previous similar message [ 5133.281276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5133.284512] Lustre: Skipped 1 previous similar message [ 5135.328215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5135.428251] Lustre: server umount lustre-MDT0000 complete [ 5137.331367] LustreError: 296066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223655 with bad export cookie 5984439679205914061 [ 5137.334364] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5137.339744] LustreError: 296066:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5137.507338] Lustre: server umount lustre-MDT0001 complete [ 5142.399419] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5143.810205] Key type lgssc unregistered [ 5143.987816] LNet: 299289:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5143.992276] LNetError: 299289:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5144.001783] LNet: Removed LNI 192.168.206.140@tcp [ 5144.422304] Key type .llcrypt unregistered [ 5144.424276] Key type ._llcrypt unregistered [ 5157.524305] Key type ._llcrypt registered [ 5157.526438] Key type .llcrypt registered [ 5157.594303] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5158.144514] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5158.158696] alg: No test for adler32 (adler32-zlib) [ 5159.052988] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 5159.171215] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5160.793204] Key type lgssc registered [ 5161.435221] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5165.741994] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5165.769905] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5167.034140] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5169.182779] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5174.128142] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5174.326880] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5176.707369] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5179.766581] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5181.916603] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5187.558629] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5187.744351] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5190.126392] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5192.505495] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5204.453718] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5204.468029] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5204.471621] Lustre: Skipped 2 previous similar messages [ 5205.361944] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.40@tcp (stopping) [ 5209.570256] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5209.574651] Lustre: Skipped 2 previous similar messages [ 5210.021306] Lustre: server umount lustre-MDT0000 complete [ 5210.463561] LustreError: 300661:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5210.476487] LustreError: 300661:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5211.902204] LustreError: 300648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223730 with bad export cookie 2687417635992022926 [ 5211.905426] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5211.917629] LustreError: 300648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 5214.689704] LustreError: 301937:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5214.689865] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5214.706758] Lustre: Skipped 2 previous similar messages [ 5214.709605] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 5216.737520] LustreError: 302817:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5216.750172] LustreError: 302817:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5218.151901] Lustre: server umount lustre-MDT0001 complete [ 5221.481108] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 5226.379875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5238.176758] LustreError: 299977:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000755a4197 x1867748398360192/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 5238.365699] LustreError: 302271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5238.429070] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5238.480226] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 5240.630412] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5244.809447] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5245.001377] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5246.051985] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 5246.957702] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5249.693164] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5251.637380] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 1 sec [ 5253.462286] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5257.386783] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 3 sec [ 5260.258450] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5260.264353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5260.272545] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5260.274804] Lustre: Skipped 2 previous similar messages [ 5265.676588] Lustre: server umount lustre-OST0000 complete [ 5270.497310] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5270.501338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5270.509450] Lustre: Skipped 1 previous similar message [ 5270.515077] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5270.519144] Lustre: Skipped 2 previous similar messages [ 5273.838771] Lustre: server umount lustre-MDT0000 complete [ 5275.848805] LustreError: 300648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223793 with bad export cookie 2687417635992030094 [ 5275.852200] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5275.861128] LustreError: 300648:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5276.049905] Lustre: server umount lustre-MDT0001 complete [ 5281.141572] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5282.557492] Key type lgssc unregistered [ 5282.727409] LNet: 306362:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5282.733161] LNetError: 306362:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5282.746300] LNet: Removed LNI 192.168.206.140@tcp [ 5283.174387] Key type .llcrypt unregistered [ 5283.176613] Key type ._llcrypt unregistered [ 5294.898955] Key type ._llcrypt registered [ 5294.902796] Key type .llcrypt registered [ 5294.957085] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5295.350592] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5295.361805] alg: No test for adler32 (adler32-zlib) [ 5296.290355] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 5296.426428] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5298.024204] Key type lgssc registered [ 5298.637291] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5302.975258] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5302.989758] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5304.265116] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5306.008463] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5310.293240] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5310.491426] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5312.467257] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5315.190320] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5316.921711] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5319.650031] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5319.657613] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5319.661878] Lustre: Skipped 1 previous similar message [ 5319.668750] Lustre: Skipped 1 previous similar message [ 5320.673136] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5320.675661] Lustre: Skipped 1 previous similar message [ 5324.575702] Lustre: server umount lustre-MDT0000 complete [ 5325.793969] LustreError: 307738:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5325.801052] LustreError: 307738:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5326.470632] LustreError: 309128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223844 with bad export cookie 5589078042177922899 [ 5326.471621] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5326.477961] LustreError: 309128:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5326.650196] Lustre: server umount lustre-MDT0001 complete [ 5334.153942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5334.296536] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5334.314663] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5334.410417] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5336.201955] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5340.246061] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5340.315072] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5340.331543] Lustre: 310095:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5340.471413] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5342.324748] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5344.887104] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5346.716493] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5352.297222] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5352.430401] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 5352.437363] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 5352.442497] LustreError: 311753:0:(tgt_mount.c:2566:server_fill_super()) Unable to start targets: -2 [ 5352.447676] LustreError: 311753:0:(tgt_mount.c:2084:server_put_super()) no obd lustre-OST0000 [ 5352.451219] LustreError: 311753:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 5352.505776] Lustre: server umount lustre-OST0000 complete [ 5352.509502] LustreError: 311753:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 5358.888774] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5358.988756] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5359.089537] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5362.041484] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5364.776379] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 5364.840805] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5368.499498] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5368.554799] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5368.608652] Lustre: lustre-OST0001: new disk, initializing [ 5368.611827] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5368.615786] Lustre: 313179:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5368.669424] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5371.383288] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5372.956137] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5372.960690] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5372.995383] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5373.783487] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 5378.022301] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5378.022936] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5378.026016] LustreError: Skipped 1 previous similar message [ 5378.032123] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5378.035067] Lustre: Skipped 1 previous similar message [ 5381.399566] Lustre: server umount lustre-OST0001 complete [ 5383.141290] LustreError: 312219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5383.149819] LustreError: 312219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5385.185919] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5385.192971] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5385.200040] Lustre: Skipped 1 previous similar message [ 5385.205424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5385.209606] Lustre: Skipped 1 previous similar message [ 5388.264937] LustreError: 312219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5388.279747] LustreError: 312219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5389.606657] Lustre: server umount lustre-OST0000 complete [ 5391.842087] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5391.846635] LustreError: Skipped 1 previous similar message [ 5391.849641] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5391.856569] Lustre: Skipped 1 previous similar message [ 5391.860836] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5391.865101] Lustre: Skipped 2 previous similar messages [ 5403.617477] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5403.623379] Lustre: Skipped 5 previous similar messages [ 5406.176137] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5406.289522] Lustre: server umount lustre-MDT0000 complete [ 5408.302770] LustreError: 310820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781223926 with bad export cookie 5589078042177923438 [ 5408.306278] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5408.309985] LustreError: 310820:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5408.498136] Lustre: server umount lustre-MDT0001 complete [ 5413.355312] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5414.764697] Key type lgssc unregistered [ 5414.924580] LNet: 315004:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5414.928044] LNetError: 315004:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5414.938426] LNet: Removed LNI 192.168.206.140@tcp [ 5415.374857] Key type .llcrypt unregistered [ 5415.377409] Key type ._llcrypt unregistered [ 5460.150932] Key type ._llcrypt registered [ 5460.152753] Key type .llcrypt registered [ 5460.213478] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 5468.227197] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5468.717663] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5468.742724] alg: No test for adler32 (adler32-zlib) [ 5469.663436] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 5469.790248] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5471.392290] Key type lgssc registered [ 5471.975714] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5488.737717] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5488.748092] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5489.899867] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5489.919130] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5489.967282] Lustre: lustre-MDT0000: new disk, initializing [ 5490.016076] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5490.031558] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5492.052426] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5497.494294] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5497.555241] Lustre: 319874:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5497.590751] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5497.593453] Lustre: Skipped 1 previous similar message [ 5497.641973] Lustre: lustre-MDT0001: new disk, initializing [ 5497.688409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5497.709702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5497.716100] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5499.333349] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5503.046300] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5504.578400] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5507.733274] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5507.862847] Lustre: lustre-OST0000: new disk, initializing [ 5507.865506] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5507.869703] Lustre: 321618:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5507.914468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5509.768244] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5509.777401] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5509.828723] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5510.621563] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5514.701588] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5516.975083] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5517.091503] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5519.172395] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5519.303220] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5525.476101] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5525.476356] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5525.482625] LustreError: Skipped 1 previous similar message [ 5525.489341] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5525.493925] Lustre: Skipped 1 previous similar message [ 5526.509630] Lustre: server umount lustre-OST0000 complete [ 5528.547000] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5528.551239] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5528.558787] Lustre: Skipped 1 previous similar message [ 5528.562348] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5530.594024] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5530.594803] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5530.606016] Lustre: Skipped 1 previous similar message [ 5534.513680] Lustre: server umount lustre-MDT0000 complete [ 5536.472508] LustreError: 321617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224054 with bad export cookie 14866987705816798635 [ 5536.477074] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5536.482511] LustreError: 321617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5536.664193] Lustre: server umount lustre-MDT0001 complete [ 5540.990563] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 20:27:39 (1781224059) [ 5545.167648] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 20:27:43 (1781224063) [ 5549.934136] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5550.173701] LustreError: 324222:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5550.183870] LustreError: 324222:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5550.213762] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5552.250221] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5555.681795] LustreError: 324223:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5556.556150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5556.758554] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5558.763669] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5561.482443] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5563.339322] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5566.950865] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5567.125469] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5569.713250] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5572.408674] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5587.140936] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 5588.217924] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5589.065746] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 5590.008982] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 5590.980400] Lustre: Failing over lustre-MDT0000 [ 5591.143951] Lustre: server umount lustre-MDT0000 complete [ 5592.035875] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5592.039189] LustreError: 324219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5592.044069] Lustre: Skipped 2 previous similar messages [ 5592.051620] LustreError: 324219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5594.480204] LustreError: 324219:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5594.491182] LustreError: 324219:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 5599.589308] LustreError: 324217:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5599.598997] LustreError: 324217:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 5605.821431] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5605.897990] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5606.065540] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5606.103616] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5608.234682] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5609.827569] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5611.497918] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5611.518569] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 5611.545577] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 5613.542381] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5614.322369] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5616.588121] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 5617.478944] Lustre: Failing over lustre-MDT0000 [ 5617.606394] Lustre: server umount lustre-MDT0000 complete [ 5620.078290] LustreError: 324218:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.40@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5620.086258] LustreError: 324218:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 5621.730264] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5621.730557] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5621.748317] Lustre: Skipped 1 previous similar message [ 5632.496064] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5632.570331] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5632.739461] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 5633.397035] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 5635.214543] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5638.134297] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5638.141970] Lustre: Skipped 2 previous similar messages [ 5638.167240] Lustre: lustre-MDT0000: Recovery over after 0:05, of 2 clients 2 recovered and 0 were evicted. [ 5638.192651] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 5640.256293] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5641.136963] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 5648.357907] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5648.358597] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5648.359371] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5648.366453] Lustre: Skipped 3 previous similar messages [ 5650.741395] Lustre: server umount lustre-MDT0000 complete [ 5652.521566] LustreError: 324202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224170 with bad export cookie 14866987705816801288 [ 5652.525503] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5652.529402] LustreError: 324202:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5652.679823] Lustre: server umount lustre-MDT0001 complete [ 5664.805972] Lustre: server umount lustre-OST0000 complete [ 5672.412423] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5672.683201] LustreError: 330002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5672.695422] LustreError: 330002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 11 previous similar messages [ 5672.726329] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5672.730738] Lustre: Skipped 1 previous similar message [ 5675.214595] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5680.503206] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5682.879131] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5685.775344] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5687.718774] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5691.659201] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5694.970529] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5696.941556] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 5698.398571] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5712.354730] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5712.361964] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5712.376784] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5712.383631] Lustre: Skipped 2 previous similar messages [ 5720.545396] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5720.548287] Lustre: Skipped 2 previous similar messages [ 5726.176172] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5726.251915] Lustre: server umount lustre-OST0000 complete [ 5734.640485] Lustre: server umount lustre-MDT0000 complete [ 5735.907039] LustreError: 330002:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5735.916976] LustreError: 330002:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5736.332350] LustreError: 329985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224254 with bad export cookie 14866987705816802037 [ 5736.335034] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5736.337277] LustreError: 329985:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5736.513328] Lustre: server umount lustre-MDT0001 complete [ 5741.425160] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5742.756294] Key type lgssc unregistered [ 5742.919735] LNet: 333198:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5742.925812] LNetError: 333198:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5742.940972] LNet: Removed LNI 192.168.206.140@tcp [ 5743.370519] Key type .llcrypt unregistered [ 5743.372469] Key type ._llcrypt unregistered [ 5755.644997] Key type ._llcrypt registered [ 5755.646732] Key type .llcrypt registered [ 5755.702740] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5756.195906] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5756.207137] alg: No test for adler32 (adler32-zlib) [ 5757.097645] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 5757.203493] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5758.800626] Key type lgssc registered [ 5759.320881] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5762.843694] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5762.853563] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5764.083275] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5765.924302] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5769.995371] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5770.173173] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5772.207388] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5774.949798] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5777.137736] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5780.868588] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5781.082989] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5784.146109] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5787.109324] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5789.293776] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 5806.050177] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5806.052301] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5806.052623] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5806.057739] Lustre: Skipped 2 previous similar messages [ 5806.062418] Lustre: Skipped 1 previous similar message [ 5808.059802] Lustre: server umount lustre-MDT0000 complete [ 5809.753596] LustreError: 334656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224327 with bad export cookie 7638510021691009997 [ 5809.757492] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5809.759179] LustreError: 334656:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5809.892784] Lustre: server umount lustre-MDT0001 complete [ 5821.060962] Lustre: server umount lustre-OST0000 complete [ 5828.491455] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5828.777838] LustreError: 338061:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5828.790155] LustreError: 338061:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5828.836509] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5831.896649] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5834.211431] LustreError: 338062:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5836.566883] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5836.764690] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5839.220275] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5842.304350] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5844.069415] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5847.147594] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5847.296163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5849.738463] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5852.180642] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5853.480812] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 5866.980511] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5866.981718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5866.986920] Lustre: Skipped 2 previous similar messages [ 5866.989450] Lustre: Skipped 1 previous similar message [ 5871.467570] Lustre: server umount lustre-MDT0000 complete [ 5872.097877] LustreError: 338057:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5872.110262] LustreError: 338057:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 5873.098869] LustreError: 338041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224391 with bad export cookie 7638510021691011096 [ 5873.100089] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5873.103144] LustreError: 338041:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5873.237783] Lustre: server umount lustre-MDT0001 complete [ 5885.170739] Lustre: server umount lustre-OST0000 complete [ 5894.119594] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5894.414328] LustreError: 341631:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5894.466924] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5896.751463] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5899.749367] LustreError: 341632:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5901.326942] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5903.477219] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5906.092133] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 5907.807610] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 5910.908685] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5913.516583] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 5915.875889] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 5919.271555] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 5924.324820] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5924.325232] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5924.330402] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5924.335492] Lustre: Skipped 2 previous similar messages [ 5928.549490] Lustre: server umount lustre-OST0000 complete [ 5932.512920] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5932.516284] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5932.521255] Lustre: Skipped 1 previous similar message [ 5932.523373] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5936.321804] Lustre: server umount lustre-MDT0000 complete [ 5938.310704] LustreError: 341612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224456 with bad export cookie 7638510021691012111 [ 5938.311962] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5938.317415] LustreError: 341612:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5938.497552] Lustre: server umount lustre-MDT0001 complete [ 5943.646431] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5945.006690] Key type lgssc unregistered [ 5945.174639] LNet: 344806:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5945.178321] LNetError: 344806:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5945.195785] LNet: Removed LNI 192.168.206.140@tcp [ 5945.594377] Key type .llcrypt unregistered [ 5945.596453] Key type ._llcrypt unregistered [ 5958.060930] Key type ._llcrypt registered [ 5958.062564] Key type .llcrypt registered [ 5958.117045] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5958.492278] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5958.514108] alg: No test for adler32 (adler32-zlib) [ 5959.388236] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 5959.496616] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 5961.104172] Key type lgssc registered [ 5961.594082] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5965.140408] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5965.151059] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5965.158553] LustreError: 346206:0:(osd_handler.c:8551:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 5965.170973] LustreError: 346206:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 5965.174770] LustreError: 346206:0:(obd_mount.c:250:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 5965.179035] LustreError: 346206:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 5965.184458] LustreError: 346206:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5968.974929] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5968.978619] LustreError: 346603:0:(osd_handler.c:8551:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 5968.989468] LustreError: 346603:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 5968.992734] LustreError: 346603:0:(obd_mount.c:250:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 5968.996276] LustreError: 346603:0:(tgt_mount.c:2534:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 5969.001082] LustreError: 346603:0:(super25.c:184:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 5977.436190] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 5978.726947] Key type lgssc unregistered [ 5978.888519] LNet: 347586:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5978.891589] LNetError: 347586:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5978.904655] LNet: Removed LNI 192.168.206.140@tcp [ 5979.268866] Key type .llcrypt unregistered [ 5979.272120] Key type ._llcrypt unregistered [ 5988.012781] Key type ._llcrypt registered [ 5988.014361] Key type .llcrypt registered [ 5988.068051] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 5997.161109] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 5997.575358] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5997.586769] alg: No test for adler32 (adler32-zlib) [ 5998.528210] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 5998.661266] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 6000.296404] Key type lgssc registered [ 6000.981780] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6019.547335] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6019.555775] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6020.738216] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6020.771467] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6020.833616] Lustre: lustre-MDT0000: new disk, initializing [ 6020.895337] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6020.905379] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6022.987422] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6029.046458] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6029.107057] Lustre: 350684:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6029.142412] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6029.146260] Lustre: Skipped 1 previous similar message [ 6029.201284] Lustre: lustre-MDT0001: new disk, initializing [ 6029.277050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6029.309088] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6029.319263] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6031.434874] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6035.940238] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6037.791479] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6041.356128] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6041.507292] Lustre: lustre-OST0000: new disk, initializing [ 6041.510618] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6041.515450] Lustre: 352426:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6041.572783] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6044.741958] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6044.837451] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6044.843615] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6044.922105] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6049.366571] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6051.440251] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6051.543146] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 6053.696670] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6053.834869] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6055.395184] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6055.395422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6055.396458] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6055.400439] LustreError: Skipped 1 previous similar message [ 6055.406463] Lustre: Skipped 1 previous similar message [ 6060.516306] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6060.524511] Lustre: Skipped 2 previous similar messages [ 6065.644937] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6065.651022] Lustre: Skipped 1 previous similar message [ 6069.216395] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6069.315448] Lustre: server umount lustre-OST0000 complete [ 6075.362763] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6075.368203] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6075.383068] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6077.763412] Lustre: server umount lustre-MDT0000 complete [ 6079.721407] LustreError: 353819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224597 with bad export cookie 14524216914551443285 [ 6079.724680] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6079.727742] LustreError: 353819:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6079.880553] Lustre: server umount lustre-MDT0001 complete [ 6084.031915] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 6084.866295] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 20:36:43 (1781224603) [ 6088.014519] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6088.276186] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6090.173257] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6093.282206] LustreError: 354737:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6093.292260] LustreError: 354737:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6094.415247] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6094.581126] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6096.491754] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6099.360395] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6101.325389] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6104.964860] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6105.171468] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6108.454372] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6111.614710] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6115.139469] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6115.233693] Lustre: lustre-OST0001: new disk, initializing [ 6115.236178] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6115.239497] Lustre: 357229:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6118.168652] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6122.014522] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6122.020212] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6122.074721] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6122.310943] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid [ 6127.074907] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6127.078376] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6127.082934] Lustre: Skipped 1 previous similar message [ 6127.087323] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6127.092942] Lustre: Skipped 2 previous similar messages [ 6130.778679] Lustre: server umount lustre-OST0001 complete [ 6132.206105] LustreError: 357667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6132.216945] LustreError: 357667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6137.315813] LustreError: 356328:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6137.323753] LustreError: 356328:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6142.439731] LustreError: 356330:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6142.451500] LustreError: 356330:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6144.481728] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6144.488267] LustreError: Skipped 1 previous similar message [ 6144.491680] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6144.500928] Lustre: Skipped 1 previous similar message [ 6144.504585] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6144.508845] Lustre: Skipped 1 previous similar message [ 6147.556273] LustreError: 356331:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6147.564252] LustreError: 356331:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6150.631897] Lustre: server umount lustre-OST0000 complete [ 6152.675271] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6152.681739] Lustre: Skipped 2 previous similar messages [ 6158.698645] Lustre: server umount lustre-MDT0000 complete [ 6160.606873] LustreError: 357228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224678 with bad export cookie 14524216914551444167 [ 6160.614076] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6160.616234] LustreError: 357228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6166.817511] Lustre: server umount lustre-MDT0001 complete [ 6171.897292] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6173.316277] Key type lgssc unregistered [ 6173.464137] LNet: 359084:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6173.470276] LNetError: 359084:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6173.484428] LNet: Removed LNI 192.168.206.140@tcp [ 6173.841207] Key type .llcrypt unregistered [ 6173.843113] Key type ._llcrypt unregistered [ 6200.843617] Key type ._llcrypt registered [ 6200.845355] Key type .llcrypt registered [ 6200.892173] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6201.384597] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6201.393278] alg: No test for adler32 (adler32-zlib) [ 6202.276741] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6202.386067] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 6203.985575] Key type lgssc registered [ 6204.530463] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6208.543785] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6208.557219] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6209.725841] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6209.739369] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6209.845240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6211.766053] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6215.640925] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6215.696808] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6215.709700] Lustre: 361842:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6215.844117] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6217.783365] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6220.528759] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6222.273813] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6223.841478] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6223.842086] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6223.846832] Lustre: Skipped 1 previous similar message [ 6228.961236] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6228.964696] Lustre: Skipped 2 previous similar messages [ 6229.413639] Lustre: server umount lustre-MDT0000 complete [ 6230.926194] LustreError: 362566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224749 with bad export cookie 5986441013645233507 [ 6230.930782] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6230.932131] LustreError: 362566:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6231.063494] Lustre: server umount lustre-MDT0001 complete [ 6235.485673] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 20:39:13 (1781224753) [ 6238.606308] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6238.856308] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6240.681824] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6244.327525] LustreError: 364126:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6244.338600] LustreError: 364126:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6245.119434] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6245.295790] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6247.204568] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6249.905959] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6251.621827] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6255.414435] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6255.521781] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6255.615645] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6258.761788] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6261.389745] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6272.536804] Lustre: Permanently deactivating lustre-OST0000 [ 6272.550375] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 6278.144975] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6279.390281] LustreError: 366770:0:(mgs_llog.c:2031:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [ 6279.398268] LustreError: 366770:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 6279.402993] LustreError: 366770:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6280.666669] LustreError: 366818:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 6280.671894] LustreError: 366818:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6287.782711] Lustre: server umount lustre-OST0000 complete [ 6289.101900] LustreError: 367067:0:(mgs_llog.c:2086:mgs_replace_nids()) Only MGS is allowed to be started [ 6289.105895] LustreError: 367067:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [ 6290.915260] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6290.922294] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6296.034216] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6296.039552] Lustre: Skipped 2 previous similar messages [ 6300.166125] Lustre: server umount lustre-MDT0000 complete [ 6301.157183] LustreError: 364834:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6301.169444] LustreError: 364834:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6302.209211] LustreError: 366475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224820 with bad export cookie 5986441013645234067 [ 6302.213723] LustreError: 366475:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6302.213846] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6312.506660] Lustre: server umount lustre-MDT0001 complete [ 6316.819412] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6318.821540] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6320.304258] LustreError: 368210:0:(mgs_handler.c:1244:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [ 6323.628641] Lustre: 368548:0:(mgs_llog.c:1798:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.140@tcp [ 6324.662584] Lustre: server umount MGS complete [ 6330.003466] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6330.279996] LustreError: 369271:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6330.291727] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6330.294812] Lustre: Skipped 1 previous similar message [ 6330.328354] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6332.268724] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6335.457293] LustreError: 369272:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6336.651557] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6336.811805] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 6338.959660] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6341.682954] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6343.686098] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6345.097803] Lustre: Permanently reactivating lustre-OST0000 [ 6345.112542] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 6345.118038] Lustre: Skipped 2 previous similar messages [ 6372.320208] LustreError: 370841:0:(osp_dev.c:693:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [ 6376.657678] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6376.856907] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6376.863715] Lustre: Skipped 1 previous similar message [ 6380.017242] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6382.548287] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6391.265046] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6391.273501] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6391.281959] Lustre: Skipped 1 previous similar message [ 6391.284632] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6392.293277] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6395.754375] Lustre: server umount lustre-OST0000 complete [ 6398.433725] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6398.438803] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6398.445336] Lustre: Skipped 1 previous similar message [ 6402.532444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6402.533069] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6402.535207] Lustre: Skipped 3 previous similar messages [ 6403.689877] Lustre: server umount lustre-MDT0000 complete [ 6405.417597] LustreError: 369252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781224923 with bad export cookie 5986441013645235124 [ 6405.420306] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6405.422181] LustreError: 369252:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6405.572579] Lustre: server umount lustre-MDT0001 complete [ 6410.112636] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6411.535642] Key type lgssc unregistered [ 6411.696660] LNet: 373420:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6411.701141] LNetError: 373420:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6411.708548] LNet: Removed LNI 192.168.206.140@tcp [ 6412.134724] Key type .llcrypt unregistered [ 6412.136672] Key type ._llcrypt unregistered [ 6420.042960] Key type ._llcrypt registered [ 6420.044105] Key type .llcrypt registered [ 6420.094576] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 6427.512945] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6427.925154] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6427.975018] alg: No test for adler32 (adler32-zlib) [ 6428.872300] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6428.999760] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 6430.608170] Key type lgssc registered [ 6431.202158] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6446.706129] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6446.716351] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6447.825774] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6447.839956] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6447.878660] Lustre: lustre-MDT0000: new disk, initializing [ 6447.929160] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6447.937222] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6449.583626] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6455.176662] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6455.235503] Lustre: 376513:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6455.272557] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6455.274553] Lustre: Skipped 1 previous similar message [ 6455.313433] Lustre: lustre-MDT0001: new disk, initializing [ 6455.368298] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6455.386104] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6455.393258] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6456.987426] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6460.748381] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6462.367926] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6465.248950] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6465.389457] Lustre: lustre-OST0000: new disk, initializing [ 6465.392288] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6465.396650] Lustre: 378256:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6465.440254] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6467.933263] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6471.585466] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6473.196917] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 6473.723567] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6473.727506] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6473.765968] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6474.303623] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 1 sec [ 6475.859992] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 6475.955263] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 6478.817746] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6478.818060] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6478.821264] LustreError: Skipped 1 previous similar message [ 6478.827498] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6478.832286] Lustre: Skipped 1 previous similar message [ 6482.848463] Lustre: server umount lustre-OST0000 complete [ 6485.985494] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6485.989208] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6485.995274] Lustre: Skipped 1 previous similar message [ 6485.997980] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6489.058577] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6489.061539] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6489.070446] Lustre: Skipped 1 previous similar message [ 6490.515269] LustreError: 379655:0:(ldlm_resource.c:1180:ldlm_resource_complain()) MGS: namespace resource [0x65727473756c:0x0:0x0].0x0 (00000000c797f758) refcount nonzero (2) after lock cleanup; forcing cleanup. [ 6490.557756] Lustre: server umount lustre-MDT0000 complete [ 6492.193889] LustreError: 376506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225010 with bad export cookie 4061042886580495777 [ 6492.196119] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6492.201933] LustreError: 376506:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6492.348503] Lustre: server umount lustre-MDT0001 complete [ 6496.267165] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [ 6497.045716] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 20:43:35 (1781225015) [ 6499.889075] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6500.102520] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6501.650589] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6505.222440] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6505.394346] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6507.046576] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6509.147832] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6510.601557] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6513.373526] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6513.524297] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6516.032942] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6518.522723] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6519.390358] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [ 6530.503292] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [ 6536.162566] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6536.162934] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6536.168479] LustreError: Skipped 1 previous similar message [ 6536.169480] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6536.177109] Lustre: Skipped 1 previous similar message [ 6538.474723] Lustre: server umount lustre-OST0000 complete [ 6541.282408] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6541.282436] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6541.287237] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6541.294806] Lustre: Skipped 1 previous similar message [ 6541.298244] Lustre: Skipped 1 previous similar message [ 6546.677343] Lustre: server umount lustre-MDT0000 complete [ 6548.671920] LustreError: 380553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225066 with bad export cookie 4061042886580496645 [ 6548.674852] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6548.679115] LustreError: 380553:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6548.839655] Lustre: server umount lustre-MDT0001 complete [ 6554.940134] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6556.687218] Key type lgssc unregistered [ 6556.853546] LNet: 383957:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6556.859364] LNetError: 383957:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6556.871651] LNet: Removed LNI 192.168.206.140@tcp [ 6557.329610] Key type .llcrypt unregistered [ 6557.331722] Key type ._llcrypt unregistered [ 6567.301035] Key type ._llcrypt registered [ 6567.302918] Key type .llcrypt registered [ 6567.355446] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6567.490370] Key type .llcrypt unregistered [ 6567.492939] Key type ._llcrypt unregistered [ 6575.877582] Key type ._llcrypt registered [ 6575.879239] Key type .llcrypt registered [ 6575.922137] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6576.278714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6576.291574] alg: No test for adler32 (adler32-zlib) [ 6577.186435] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6577.292788] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 6578.896967] Key type lgssc registered [ 6579.453548] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6583.153370] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6583.162975] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6584.388188] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6586.117579] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6589.888219] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6590.044164] LustreError: 386115:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6590.049583] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6590.052756] LustreError: 386115:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6592.521851] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6594.146624] LustreError: 386119:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6594.156318] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 6594.159321] LustreError: 386119:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6594.827990] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6597.935239] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6598.113149] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6599.738406] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6608.357270] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6608.358577] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6608.358701] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6608.366438] Lustre: Skipped 1 previous similar message [ 6613.478641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6613.483336] Lustre: Skipped 2 previous similar messages [ 6614.182696] Lustre: server umount lustre-OST0000 complete [ 6618.086059] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6618.090701] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6618.097831] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6618.568402] Lustre: server umount lustre-MDT0000 complete [ 6618.601802] LustreError: 386113:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6620.959597] LustreError: 386100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225139 with bad export cookie 5712897010132649076 [ 6620.964025] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6620.964094] LustreError: 386100:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6621.159687] Lustre: server umount lustre-MDT0001 complete [ 6626.887214] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6628.367198] Key type lgssc unregistered [ 6628.541819] LNet: 389097:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6628.547312] LNetError: 389097:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6628.561390] LNet: Removed LNI 192.168.206.140@tcp [ 6628.981856] Key type .llcrypt unregistered [ 6628.983977] Key type ._llcrypt unregistered [ 6640.592362] Key type ._llcrypt registered [ 6640.596374] Key type .llcrypt registered [ 6640.660740] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6641.213257] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6641.255108] alg: No test for adler32 (adler32-zlib) [ 6642.194188] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6642.352660] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 6643.960374] Key type lgssc registered [ 6644.505734] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6648.335522] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6648.343088] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6663.840356] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 6668.960190] LustreError: 390448:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.140@tcp: failed processing log, type 4: rc = -110 [ 6695.584230] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6698.442622] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6701.112818] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6704.325482] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6704.587161] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6704.631294] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 6706.554879] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6709.730900] LustreError: 390473:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6709.738854] LustreError: 390473:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6710.761203] LustreError: 391369:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6710.772644] LustreError: 391369:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6710.938277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6711.141811] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6712.945478] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6715.422972] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6717.051977] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6721.507618] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6721.507835] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6721.508836] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6721.512992] Lustre: Skipped 1 previous similar message [ 6726.627385] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6726.631210] Lustre: Skipped 2 previous similar messages [ 6727.005835] Lustre: server umount lustre-OST0000 complete [ 6730.724386] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [ 6730.728983] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6730.736661] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6731.748248] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6734.698141] Lustre: server umount lustre-MDT0000 complete [ 6736.273664] LustreError: 390454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225254 with bad export cookie 8292359432442077838 [ 6736.275957] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6736.279114] LustreError: 390454:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6736.416407] Lustre: server umount lustre-MDT0001 complete [ 6740.749682] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6741.967384] Key type lgssc unregistered [ 6742.112627] LNet: 393659:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6742.116603] LNetError: 393659:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6742.129455] LNet: Removed LNI 192.168.206.140@tcp [ 6742.466067] Key type .llcrypt unregistered [ 6742.467981] Key type ._llcrypt unregistered [ 6753.147932] Key type ._llcrypt registered [ 6753.149165] Key type .llcrypt registered [ 6753.196950] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6753.592445] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6753.636060] alg: No test for adler32 (adler32-zlib) [ 6754.515178] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6754.614913] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 6756.216146] Key type lgssc registered [ 6756.669981] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6760.190064] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6760.205525] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6761.442452] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6763.119299] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6766.905684] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6767.110443] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6768.910915] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6771.227286] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6772.832361] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6775.913743] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6776.086771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6778.972947] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6781.569945] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6786.345063] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 6791.652850] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6791.655177] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6791.659305] Lustre: Skipped 2 previous similar messages [ 6791.662472] Lustre: Skipped 1 previous similar message [ 6794.037798] Lustre: server umount lustre-MDT0000 complete [ 6795.065507] LustreError: 395015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.40@tcp arrived at 1781225313 with bad export cookie 18008834187420631095 [ 6795.074323] LustreError: 395015:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6795.949540] LustreError: 395017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225314 with bad export cookie 18008834187420630444 [ 6795.954864] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6795.957622] LustreError: 395017:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6796.769240] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6796.769887] LustreError: 396645:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6796.781972] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6796.787122] LustreError: 396645:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 3 previous similar messages [ 6796.789483] Lustre: Skipped 2 previous similar messages [ 6800.801359] LustreError: 397181:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6800.802161] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6800.812183] LustreError: 397181:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6802.100683] Lustre: server umount lustre-OST0000 complete [ 6805.317285] Lustre: server umount lustre-MDT0001 complete [ 6809.995540] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6811.359163] Key type lgssc unregistered [ 6811.513652] LNet: 398361:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6811.517429] LNetError: 398361:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6811.524347] LNet: Removed LNI 192.168.206.140@tcp [ 6811.929184] Key type .llcrypt unregistered [ 6811.930925] Key type ._llcrypt unregistered [ 6823.779895] Key type ._llcrypt registered [ 6823.781611] Key type .llcrypt registered [ 6823.829962] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6824.365020] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6824.373121] alg: No test for adler32 (adler32-zlib) [ 6825.294017] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6825.396360] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 6827.000170] Key type lgssc registered [ 6827.550662] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6831.595753] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6831.608338] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6832.850936] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6834.785347] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6839.552993] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6839.780072] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6842.072705] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6844.995247] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 6847.010519] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 6850.451966] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6850.617920] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6853.752562] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6856.819754] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6857.778270] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 6860.208602] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6860.217920] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6860.225834] Lustre: Skipped 1 previous similar message [ 6864.224869] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.40@tcp (stopping) [ 6865.379760] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6865.384102] Lustre: Skipped 1 previous similar message [ 6866.419588] Lustre: server umount lustre-MDT0001 complete [ 6870.499530] LustreError: 399740:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6870.500383] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6870.513680] LustreError: 399740:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 6870.528794] Lustre: Skipped 1 previous similar message [ 6870.535697] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6875.433951] Lustre: server umount lustre-OST0000 complete [ 6875.618296] LustreError: 399739:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6877.442192] Lustre: server umount lustre-MDT0000 complete [ 6883.857731] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 6885.269503] Key type lgssc unregistered [ 6885.420995] LNet: 403062:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6885.427911] LNetError: 403062:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6885.442583] LNet: Removed LNI 192.168.206.140@tcp [ 6885.824392] Key type .llcrypt unregistered [ 6885.825912] Key type ._llcrypt unregistered [ 6897.634386] Key type ._llcrypt registered [ 6897.636155] Key type .llcrypt registered [ 6897.695884] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 6905.366666] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6905.875480] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6905.893210] alg: No test for adler32 (adler32-zlib) [ 6906.777550] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 6906.880733] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 6908.496168] Key type lgssc registered [ 6909.196851] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6928.495782] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 6936.493881] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6936.510056] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6937.668620] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6937.695723] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6937.750422] Lustre: lustre-MDT0000: new disk, initializing [ 6937.803681] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6937.816680] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6940.035699] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6948.834297] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6948.986175] Lustre: lustre-OST0000: new disk, initializing [ 6948.988742] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6948.992023] Lustre: Skipped 1 previous similar message [ 6948.996175] Lustre: 407999:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6949.029689] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6951.872455] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6954.006860] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 6954.012934] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 6954.029494] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 6955.899088] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 6965.121030] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6965.203263] Lustre: 407001:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6965.251580] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6965.300594] Lustre: lustre-MDT0001: new disk, initializing [ 6965.364167] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6965.388611] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [ 6965.393436] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [ 6967.305450] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 6972.367711] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6972.503798] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6974.321753] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6974.413628] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6978.019773] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [ 6978.021108] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6978.027797] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 6978.034579] Lustre: Skipped 1 previous similar message [ 6981.489851] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.40@tcp (stopping) [ 6981.493962] Lustre: Skipped 1 previous similar message [ 6982.765151] Lustre: server umount lustre-MDT0001 complete [ 6988.260202] LustreError: 407008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6988.268392] LustreError: 407008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 6993.379804] LustreError: 407008:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6993.392294] LustreError: 407008:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 6998.497509] LustreError: 409941:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6998.497762] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6998.520367] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7002.678407] Lustre: server umount lustre-OST0000 complete [ 7003.620798] LustreError: 407012:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7010.791891] Lustre: server umount lustre-MDT0000 complete [ 7017.441973] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 7019.047433] Key type lgssc unregistered [ 7019.214983] LNet: 411560:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7019.222190] LNetError: 411560:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7019.235823] LNet: Removed LNI 192.168.206.140@tcp [ 7019.711659] Key type .llcrypt unregistered [ 7019.714256] Key type ._llcrypt unregistered [ 7039.218445] Key type ._llcrypt registered [ 7039.221280] Key type .llcrypt registered [ 7039.280816] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 7039.430335] Key type .llcrypt unregistered [ 7039.432718] Key type ._llcrypt unregistered [ 7048.295318] Key type ._llcrypt registered [ 7048.298131] Key type .llcrypt registered [ 7048.359095] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 7048.752578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7048.828580] alg: No test for adler32 (adler32-zlib) [ 7049.755960] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 7049.883382] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 7051.505349] Key type lgssc registered [ 7052.116945] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7067.186228] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7067.204545] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7068.357757] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7068.379300] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7068.430039] Lustre: lustre-MDT0000: new disk, initializing [ 7068.487219] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7068.497722] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7070.437411] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7076.784200] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7076.873929] Lustre: 414874:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7076.911819] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7076.915269] Lustre: Skipped 1 previous similar message [ 7076.987596] Lustre: lustre-MDT0001: new disk, initializing [ 7077.049584] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7077.083545] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7077.091869] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7079.144652] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7083.460281] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7085.347351] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7089.209703] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7089.354941] Lustre: lustre-OST0000: new disk, initializing [ 7089.358052] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7089.361214] Lustre: 416617:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7089.405053] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7092.217674] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7096.428820] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7096.857045] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7096.865474] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7096.917875] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7107.044410] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7107.047459] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7107.051951] Lustre: Skipped 2 previous similar messages [ 7107.058653] Lustre: Skipped 2 previous similar messages [ 7110.790576] Lustre: server umount lustre-MDT0000 complete [ 7112.794444] LustreError: 414868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225630 with bad export cookie 4457050827469273205 [ 7112.795126] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7112.809268] LustreError: 414868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7112.961187] Lustre: server umount lustre-MDT0001 complete [ 7125.439655] Lustre: server umount lustre-OST0000 complete [ 7130.480750] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 20:54:08 (1781225648) [ 7134.213541] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7134.447141] LustreError: 418530:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7134.457113] LustreError: 418530:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 4 previous similar messages [ 7134.499469] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7136.320464] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7139.809733] LustreError: 418531:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7140.429489] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7140.652940] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7142.707645] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7145.592783] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7147.460132] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7164.518414] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7164.646897] Lustre: Found index 0 for lustre-OST0000, updating log [ 7164.753679] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7168.035993] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7171.458955] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7180.140128] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [ 7185.383346] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7185.384174] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7185.384853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7185.390804] Lustre: Skipped 1 previous similar message [ 7186.402824] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7186.407656] Lustre: Skipped 2 previous similar messages [ 7186.812409] Lustre: server umount lustre-OST0000 complete [ 7202.276274] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7202.279994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7202.287111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7206.642270] Lustre: server umount lustre-MDT0000 complete [ 7207.907442] LustreError: 418526:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7207.918966] LustreError: 418526:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 1 previous similar message [ 7208.331677] LustreError: 418510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781225726 with bad export cookie 4457050827469274598 [ 7208.334600] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7208.338308] LustreError: 418510:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7208.497340] Lustre: server umount lustre-MDT0001 complete [ 7212.498980] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 20:55:30 (1781225730) [ 7236.256442] LNet: 1 local NIs in recovery (showing 1): 192.168.206.140@tcp [ 7799.167997] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7799.353651] LustreError: 424623:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7799.383985] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7800.847176] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7804.300769] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7804.384769] Lustre: lustre-MDT0001: Not available for connect from 0@lo (not set up) [ 7804.388027] Lustre: Skipped 1 previous similar message [ 7804.454091] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7806.071020] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7808.290508] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 7809.762232] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 7825.507519] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7825.599143] Lustre: Found index 0 for lustre-OST0000, updating log [ 7825.709321] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7827.754106] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:65) [ 7828.052784] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7830.357928] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 7832.354436] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7832.358867] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7832.365267] Lustre: Skipped 1 previous similar message [ 7837.412458] Lustre: server umount lustre-OST0000 complete [ 7839.008678] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7839.014229] Lustre: Skipped 1 previous similar message [ 7845.154743] LustreError: 424642:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7845.157600] Lustre: server umount lustre-MDT0000 complete [ 7846.754603] LustreError: 424603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226364 with bad export cookie 4457050827469275669 [ 7846.755194] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7846.759596] LustreError: 424603:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 7846.921206] Lustre: server umount lustre-MDT0001 complete [ 7849.633551] LNet: 1 local NIs in recovery (showing 1): 192.168.206.140@tcp [ 7866.046096] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 7870.074364] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 7886.429635] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 21:06:44 (1781226404) [ 7895.400853] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 7896.906831] Key type lgssc unregistered [ 7897.070768] LNet: 430886:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7897.074236] LNetError: 430886:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7956.455361] LNet: Removed LNI 192.168.206.140@tcp [ 7956.845721] Key type .llcrypt unregistered [ 7956.848656] Key type ._llcrypt unregistered [ 7964.747957] Key type ._llcrypt registered [ 7964.749501] Key type .llcrypt registered [ 7964.797382] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 7965.246362] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7965.312598] alg: No test for adler32 (adler32-zlib) [ 7966.208675] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 7966.312140] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 7967.920302] Key type lgssc registered [ 7968.505285] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7970.669181] LNet: Added LNI 192.168.206.140@tcp100 [8/256/0/180] [ 7971.098781] LNet: Added LNI 192.168.206.140@tcp101 [8/256/0/180] [ 7971.529499] LNet: Added LNI 192.168.206.140@tcp102 [8/256/0/180] [ 7971.974636] LNet: Added LNI 192.168.206.140@tcp103 [8/256/0/180] [ 7972.405691] LNet: Added LNI 192.168.206.140@tcp104 [8/256/0/180] [ 7972.822891] LNet: Added LNI 192.168.206.140@tcp105 [8/256/0/180] [ 7973.248397] LNet: Added LNI 192.168.206.140@tcp106 [8/256/0/180] [ 7973.636993] LNet: Added LNI 192.168.206.140@tcp107 [8/256/0/180] [ 7974.025989] LNet: Added LNI 192.168.206.140@tcp108 [8/256/0/180] [ 7974.424882] LNet: Added LNI 192.168.206.140@tcp109 [8/256/0/180] [ 7974.790934] LNet: Added LNI 192.168.206.140@tcp110 [8/256/0/180] [ 7975.196204] LNet: Added LNI 192.168.206.140@tcp111 [8/256/0/180] [ 7975.585869] LNet: Added LNI 192.168.206.140@tcp112 [8/256/0/180] [ 7975.961407] LNet: Added LNI 192.168.206.140@tcp113 [8/256/0/180] [ 7976.338259] LNet: Added LNI 192.168.206.140@tcp114 [8/256/0/180] [ 7976.717736] LNet: Added LNI 192.168.206.140@tcp115 [8/256/0/180] [ 7977.089405] LNet: Added LNI 192.168.206.140@tcp116 [8/256/0/180] [ 7977.480219] LNet: Added LNI 192.168.206.140@tcp117 [8/256/0/180] [ 7977.873115] LNet: Added LNI 192.168.206.140@tcp118 [8/256/0/180] [ 7978.258317] LNet: Added LNI 192.168.206.140@tcp119 [8/256/0/180] [ 7978.634601] LNet: Added LNI 192.168.206.140@tcp120 [8/256/0/180] [ 7979.025935] LNet: Added LNI 192.168.206.140@tcp121 [8/256/0/180] [ 7979.418728] LNet: Added LNI 192.168.206.140@tcp122 [8/256/0/180] [ 7979.868862] LNet: Added LNI 192.168.206.140@tcp123 [8/256/0/180] [ 7980.247057] LNet: Added LNI 192.168.206.140@tcp124 [8/256/0/180] [ 7980.655615] LNet: Added LNI 192.168.206.140@tcp125 [8/256/0/180] [ 7981.051735] LNet: Added LNI 192.168.206.140@tcp126 [8/256/0/180] [ 7981.434475] LNet: Added LNI 192.168.206.140@tcp127 [8/256/0/180] [ 7981.780962] LNet: Added LNI 192.168.206.140@tcp128 [8/256/0/180] [ 7982.135734] LNet: Added LNI 192.168.206.140@tcp129 [8/256/0/180] [ 7982.520456] LNet: Added LNI 192.168.206.140@tcp130 [8/256/0/180] [ 7982.945642] LNet: Added LNI 192.168.206.140@tcp131 [8/256/0/180] [ 7983.363373] LNet: Added LNI 192.168.206.140@tcp132 [8/256/0/180] [ 7983.764756] LNet: Added LNI 192.168.206.140@tcp133 [8/256/0/180] [ 7984.138587] LNet: Added LNI 192.168.206.140@tcp134 [8/256/0/180] [ 7984.554674] LNet: Added LNI 192.168.206.140@tcp135 [8/256/0/180] [ 7987.739246] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7987.753986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7988.821148] Lustre: 433866:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.140@tcp: can't add NID 192.168.206.140@tcp130: rc = -75 [ 7988.907221] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7988.922349] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7988.963753] Lustre: lustre-MDT0000: new disk, initializing [ 7989.011298] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7989.023395] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7990.693844] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 7995.891331] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7995.959646] Lustre: 433884:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7995.996449] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7995.999091] Lustre: Skipped 1 previous similar message [ 7996.044303] Lustre: lustre-MDT0001: new disk, initializing [ 7996.061202] Lustre: 434664:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.140@tcp: can't add NID 192.168.206.140@tcp130: rc = -75 [ 7996.066161] Lustre: 434664:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 7996.129953] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7996.152677] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7996.158452] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7998.132614] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8002.400517] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8002.558344] Lustre: 435168:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.140@tcp: can't add NID 192.168.206.140@tcp130: rc = -75 [ 8002.562724] Lustre: 435168:0:(lustre_peer.c:105:class_add_uuid()) Skipped 11 previous similar messages [ 8004.383537] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8007.802785] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8007.959675] Lustre: lustre-OST0000: new disk, initializing [ 8007.963172] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8007.968743] Lustre: 435626:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8008.018177] Lustre: 435631:0:(lustre_peer.c:105:class_add_uuid()) 192.168.206.140@tcp: can't add NID 192.168.206.140@tcp130: rc = -75 [ 8008.022994] Lustre: 435631:0:(lustre_peer.c:105:class_add_uuid()) Skipped 5 previous similar messages [ 8008.045106] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8008.248955] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8008.254454] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8008.311622] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8011.042076] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8015.205542] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8018.400941] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8018.405143] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8018.411419] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8018.914806] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8018.915217] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8018.926459] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8024.037587] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8024.040727] Lustre: Skipped 1 previous similar message [ 8024.493428] Lustre: server umount lustre-OST0000 complete [ 8027.105358] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8027.108909] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8027.114870] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8029.153950] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8032.314642] Lustre: server umount lustre-MDT0000 complete [ 8034.019716] LustreError: 433875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226552 with bad export cookie 13097988635008846533 [ 8034.024083] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8034.027135] LustreError: 433875:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8034.185412] Lustre: server umount lustre-MDT0001 complete [ 8038.629882] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 8039.903214] Key type lgssc unregistered [ 8040.105152] LNet: 437407:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8040.110607] LNetError: 437407:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8040.115061] LNet: Removed LNI 192.168.206.140@tcp [ 8041.120324] LNet: Removed LNI 192.168.206.140@tcp100 [ 8041.122352] LNetError: 437407:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8041.124317] LNetError: 437407:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8041.127145] LNet: Removed LNI 192.168.206.140@tcp101 [ 8042.144176] LNet: Removed LNI 192.168.206.140@tcp102 [ 8042.146236] LNetError: 437407:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8042.148708] LNetError: 437407:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [ 8042.152276] LNet: Removed LNI 192.168.206.140@tcp103 [ 8043.169250] LNet: Removed LNI 192.168.206.140@tcp104 [ 8044.192876] LNet: Removed LNI 192.168.206.140@tcp105 [ 8044.195127] LNetError: 437407:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8044.200447] LNetError: 437407:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [ 8044.204492] LNet: Removed LNI 192.168.206.140@tcp106 [ 8045.216295] LNet: Removed LNI 192.168.206.140@tcp107 [ 8046.240175] LNet: Removed LNI 192.168.206.140@tcp108 [ 8046.243625] LNet: Removed LNI 192.168.206.140@tcp109 [ 8046.245455] LNet: Removed LNI 192.168.206.140@tcp110 [ 8046.247771] LNet: Removed LNI 192.168.206.140@tcp111 [ 8046.249929] LNet: Removed LNI 192.168.206.140@tcp112 [ 8046.252108] LNet: Removed LNI 192.168.206.140@tcp113 [ 8046.254284] LNet: Removed LNI 192.168.206.140@tcp114 [ 8046.256578] LNet: Removed LNI 192.168.206.140@tcp115 [ 8046.258868] LNet: Removed LNI 192.168.206.140@tcp116 [ 8046.261404] LNet: Removed LNI 192.168.206.140@tcp117 [ 8046.263736] LNet: Removed LNI 192.168.206.140@tcp118 [ 8046.266055] LNet: Removed LNI 192.168.206.140@tcp119 [ 8046.268446] LNet: Removed LNI 192.168.206.140@tcp120 [ 8046.270385] LNet: Removed LNI 192.168.206.140@tcp121 [ 8046.272307] LNet: Removed LNI 192.168.206.140@tcp122 [ 8046.274750] LNet: Removed LNI 192.168.206.140@tcp123 [ 8046.276918] LNet: Removed LNI 192.168.206.140@tcp124 [ 8046.278733] LNet: Removed LNI 192.168.206.140@tcp125 [ 8046.281417] LNet: Removed LNI 192.168.206.140@tcp126 [ 8046.284232] LNet: Removed LNI 192.168.206.140@tcp127 [ 8046.288142] LNet: Removed LNI 192.168.206.140@tcp128 [ 8046.290511] LNet: Removed LNI 192.168.206.140@tcp129 [ 8046.292827] LNet: Removed LNI 192.168.206.140@tcp130 [ 8046.295324] LNet: Removed LNI 192.168.206.140@tcp131 [ 8046.297334] LNet: Removed LNI 192.168.206.140@tcp132 [ 8046.299484] LNet: Removed LNI 192.168.206.140@tcp133 [ 8046.301654] LNet: Removed LNI 192.168.206.140@tcp134 [ 8046.312203] LNet: Removed LNI 192.168.206.140@tcp135 [ 8046.739113] Key type .llcrypt unregistered [ 8046.742568] Key type ._llcrypt unregistered [ 8056.599566] Key type ._llcrypt registered [ 8056.601111] Key type .llcrypt registered [ 8056.652843] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 8064.069886] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8064.439846] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8064.456837] alg: No test for adler32 (adler32-zlib) [ 8065.331757] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 8065.428709] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 8067.024167] Key type lgssc registered [ 8067.574233] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8082.189101] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 21:10:00 (1781226600) [ 8089.509532] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 8090.812218] Key type lgssc unregistered [ 8090.976596] LNet: 441411:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8090.982780] LNetError: 441411:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8090.996786] LNet: Removed LNI 192.168.206.140@tcp [ 8091.359578] Key type .llcrypt unregistered [ 8091.362454] Key type ._llcrypt unregistered [ 8099.395732] Key type ._llcrypt registered [ 8099.397446] Key type .llcrypt registered [ 8099.464700] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8099.969434] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8099.988067] alg: No test for adler32 (adler32-zlib) [ 8100.900393] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 8101.029841] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 8102.632166] Key type lgssc registered [ 8103.169671] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8107.779620] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8107.792491] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8108.923631] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8108.948490] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8108.994252] Lustre: lustre-MDT0000: new disk, initializing [ 8109.030704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8109.040438] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8110.793222] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8116.847194] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8116.913204] Lustre: 442584:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8116.948343] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8116.951569] Lustre: Skipped 1 previous similar message [ 8117.006188] Lustre: lustre-MDT0001: new disk, initializing [ 8117.069664] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8117.090352] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8117.095636] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8119.227106] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8123.028186] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8124.633521] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8127.428124] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8127.577778] Lustre: lustre-OST0000: new disk, initializing [ 8127.580368] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8127.583703] Lustre: 444323:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8127.631944] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8130.349697] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8132.623268] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8132.629621] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8132.681936] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8135.124708] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8136.599391] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing lnet_if_list [ 8138.826677] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [ 8138.842949] LNet: Added LNI 192.168.206.140@tcp10 [8/256/0/180] [ 8141.213925] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [ 8141.228065] LNet: Added LNI 192.168.206.140@tcp11 [8/256/0/180] [ 8142.779557] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [ 8142.798928] LNet: Added LNI 192.168.206.140@tcp12 [8/256/0/180] [ 8144.632969] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [ 8144.653307] LNet: Added LNI 192.168.206.140@tcp13 [8/256/0/180] [ 8146.566427] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [ 8146.580677] LNet: Added LNI 192.168.206.140@tcp14 [8/256/0/180] [ 8148.728797] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [ 8148.747454] LNet: Added LNI 192.168.206.140@tcp15 [8/256/0/180] [ 8150.820581] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [ 8150.838454] LNet: Added LNI 192.168.206.140@tcp16 [8/256/0/180] [ 8153.041897] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [ 8153.077070] LNet: Added LNI 192.168.206.140@tcp17 [8/256/0/180] [ 8155.525410] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [ 8155.544341] LNet: Added LNI 192.168.206.140@tcp18 [8/256/0/180] [ 8157.951586] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [ 8157.970909] LNet: Added LNI 192.168.206.140@tcp19 [8/256/0/180] [ 8160.235471] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [ 8160.256143] LNet: Added LNI 192.168.206.140@tcp20 [8/256/0/180] [ 8162.845106] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [ 8162.865561] LNet: Added LNI 192.168.206.140@tcp21 [8/256/0/180] [ 8165.887466] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [ 8165.904258] LNet: Added LNI 192.168.206.140@tcp22 [8/256/0/180] [ 8168.351627] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [ 8168.369474] LNet: Added LNI 192.168.206.140@tcp23 [8/256/0/180] [ 8170.901443] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [ 8170.914381] LNet: Added LNI 192.168.206.140@tcp24 [8/256/0/180] [ 8173.333354] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [ 8173.356878] LNet: Added LNI 192.168.206.140@tcp25 [8/256/0/180] [ 8175.839977] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [ 8175.853656] LNet: Added LNI 192.168.206.140@tcp26 [8/256/0/180] [ 8178.056944] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [ 8178.072890] LNet: Added LNI 192.168.206.140@tcp27 [8/256/0/180] [ 8180.416092] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [ 8180.436380] LNet: Added LNI 192.168.206.140@tcp28 [8/256/0/180] [ 8182.810522] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [ 8182.826615] LNet: Added LNI 192.168.206.140@tcp29 [8/256/0/180] [ 8185.268573] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [ 8185.284366] LNet: Added LNI 192.168.206.140@tcp30 [8/256/0/180] [ 8188.084880] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [ 8188.101064] LNet: Added LNI 192.168.206.140@tcp31 [8/256/0/180] [ 8190.641218] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [ 8190.658459] LNet: Added LNI 192.168.206.140@tcp32 [8/256/0/180] [ 8193.311709] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [ 8193.329594] LNet: Added LNI 192.168.206.140@tcp33 [8/256/0/180] [ 8196.236373] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [ 8196.250841] LNet: Added LNI 192.168.206.140@tcp34 [8/256/0/180] [ 8198.976420] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [ 8198.992560] LNet: Added LNI 192.168.206.140@tcp35 [8/256/0/180] [ 8201.884197] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [ 8201.900916] LNet: Added LNI 192.168.206.140@tcp36 [8/256/0/180] [ 8204.813301] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [ 8204.834213] LNet: Added LNI 192.168.206.140@tcp37 [8/256/0/180] [ 8207.874127] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [ 8207.888447] LNet: Added LNI 192.168.206.140@tcp38 [8/256/0/180] [ 8210.601955] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [ 8210.623969] LNet: Added LNI 192.168.206.140@tcp39 [8/256/0/180] [ 8213.478886] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [ 8213.495242] LNet: Added LNI 192.168.206.140@tcp40 [8/256/0/180] [ 8216.409075] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [ 8216.424680] LNet: Added LNI 192.168.206.140@tcp41 [8/256/0/180] [ 8219.236295] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [ 8219.251132] LNet: Added LNI 192.168.206.140@tcp42 [8/256/0/180] [ 8222.163666] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [ 8222.180216] LNet: Added LNI 192.168.206.140@tcp43 [8/256/0/180] [ 8225.425933] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [ 8225.443745] LNet: Added LNI 192.168.206.140@tcp44 [8/256/0/180] [ 8228.813716] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [ 8228.828829] LNet: Added LNI 192.168.206.140@tcp45 [8/256/0/180] [ 8231.899222] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [ 8231.915361] LNet: Added LNI 192.168.206.140@tcp46 [8/256/0/180] [ 8235.158111] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [ 8235.173791] LNet: Added LNI 192.168.206.140@tcp47 [8/256/0/180] [ 8238.539327] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [ 8238.555776] LNet: Added LNI 192.168.206.140@tcp48 [8/256/0/180] [ 8242.310791] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [ 8242.325145] LNet: Added LNI 192.168.206.140@tcp49 [8/256/0/180] [ 8246.005565] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [ 8246.023309] LNet: Added LNI 192.168.206.140@tcp50 [8/256/0/180] [ 8255.460545] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8255.460724] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8255.461673] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8255.473513] Lustre: Skipped 2 previous similar messages [ 8260.578295] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8260.584020] Lustre: Skipped 4 previous similar messages [ 8261.346487] Lustre: server umount lustre-MDT0000 complete [ 8264.477372] LustreError: 442575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226782 with bad export cookie 5414082800876551065 [ 8264.480409] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8264.483763] LustreError: 442575:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8264.615279] Lustre: server umount lustre-MDT0001 complete [ 8273.046950] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8276.661416] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8281.056181] Lustre: 441857:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226783/real 1781226783] req@00000000819cdb83 x1867751483339136/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781226799 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8281.067857] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8285.152128] Lustre: 441858:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226787/real 1781226787] req@0000000071412dfd x1867751483339264/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781226803 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8286.176235] Lustre: 441859:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226788/real 1781226788] req@0000000057076bb5 x1867751483339392/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781226804 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8289.248554] LustreError: 441855:0:(client.c:1390:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000688358f9 x1867751483340160/t0(0) o250->MGC192.168.206.140@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8289.459344] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8290.209510] Lustre: 441857:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226792/real 1781226792] req@000000002bfbf70b x1867751483339648/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781226808 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8291.492741] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8296.122915] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8296.359657] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8297.376574] Lustre: 441857:0:(client.c:2480:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1781226798/real 1781226798] req@00000000668079ee x1867751483339776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1781226814 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 8298.341804] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8301.013259] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8301.541767] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8302.662587] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8310.757026] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8311.778071] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8311.781096] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8311.788152] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8315.361390] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8315.364712] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8315.371312] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8317.418824] Lustre: server umount lustre-OST0000 complete [ 8320.994403] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8321.000807] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8321.004102] Lustre: Skipped 2 previous similar messages [ 8325.215873] Lustre: server umount lustre-MDT0000 complete [ 8326.113811] LustreError: 467667:0:(ldlm_lib.c:1179:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8326.120758] LustreError: 467667:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 6 previous similar messages [ 8326.865323] LustreError: 444322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226844 with bad export cookie 5414082800876553270 [ 8326.868314] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8326.870248] LustreError: 444322:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8327.025798] Lustre: server umount lustre-MDT0001 complete [ 8331.807491] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 8333.091975] Key type lgssc unregistered [ 8333.296135] LNet: 470204:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8333.300419] LNetError: 470204:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8333.303135] LNet: Removed LNI 192.168.206.140@tcp [ 8333.304856] LNet: Removed LNI 192.168.206.140@tcp10 [ 8333.307065] LNet: Removed LNI 192.168.206.140@tcp11 [ 8333.309154] LNet: Removed LNI 192.168.206.140@tcp12 [ 8333.311171] LNet: Removed LNI 192.168.206.140@tcp13 [ 8333.313177] LNet: Removed LNI 192.168.206.140@tcp14 [ 8333.314910] LNet: Removed LNI 192.168.206.140@tcp15 [ 8333.316500] LNet: Removed LNI 192.168.206.140@tcp16 [ 8333.317902] LNet: Removed LNI 192.168.206.140@tcp17 [ 8333.320886] LNet: Removed LNI 192.168.206.140@tcp18 [ 8333.323062] LNet: Removed LNI 192.168.206.140@tcp19 [ 8333.325386] LNet: Removed LNI 192.168.206.140@tcp20 [ 8333.328197] LNet: Removed LNI 192.168.206.140@tcp21 [ 8333.330568] LNet: Removed LNI 192.168.206.140@tcp22 [ 8333.332807] LNet: Removed LNI 192.168.206.140@tcp23 [ 8333.334926] LNet: Removed LNI 192.168.206.140@tcp24 [ 8333.336597] LNet: Removed LNI 192.168.206.140@tcp25 [ 8333.338855] LNet: Removed LNI 192.168.206.140@tcp26 [ 8333.340852] LNet: Removed LNI 192.168.206.140@tcp27 [ 8333.343178] LNet: Removed LNI 192.168.206.140@tcp28 [ 8333.345325] LNet: Removed LNI 192.168.206.140@tcp29 [ 8333.347623] LNet: Removed LNI 192.168.206.140@tcp30 [ 8333.350177] LNet: Removed LNI 192.168.206.140@tcp31 [ 8333.352817] LNet: Removed LNI 192.168.206.140@tcp32 [ 8333.354896] LNet: Removed LNI 192.168.206.140@tcp33 [ 8333.356956] LNet: Removed LNI 192.168.206.140@tcp34 [ 8333.359590] LNet: Removed LNI 192.168.206.140@tcp35 [ 8333.361547] LNet: Removed LNI 192.168.206.140@tcp36 [ 8333.363468] LNet: Removed LNI 192.168.206.140@tcp37 [ 8333.365645] LNet: Removed LNI 192.168.206.140@tcp38 [ 8333.367202] LNet: Removed LNI 192.168.206.140@tcp39 [ 8333.369085] LNet: Removed LNI 192.168.206.140@tcp40 [ 8333.372419] LNet: Removed LNI 192.168.206.140@tcp41 [ 8333.374802] LNet: Removed LNI 192.168.206.140@tcp42 [ 8333.376632] LNet: Removed LNI 192.168.206.140@tcp43 [ 8333.379057] LNet: Removed LNI 192.168.206.140@tcp44 [ 8333.381143] LNet: Removed LNI 192.168.206.140@tcp45 [ 8333.383838] LNet: Removed LNI 192.168.206.140@tcp46 [ 8333.385363] LNet: Removed LNI 192.168.206.140@tcp47 [ 8333.387368] LNet: Removed LNI 192.168.206.140@tcp48 [ 8333.388978] LNet: Removed LNI 192.168.206.140@tcp49 [ 8333.399615] LNet: Removed LNI 192.168.206.140@tcp50 [ 8333.729339] Key type .llcrypt unregistered [ 8333.731052] Key type ._llcrypt unregistered [ 8344.516842] Key type ._llcrypt registered [ 8344.518821] Key type .llcrypt registered [ 8344.578928] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 8352.402289] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8352.847679] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8352.859946] alg: No test for adler32 (adler32-zlib) [ 8353.749890] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 8353.892042] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 8355.488670] Key type lgssc registered [ 8355.925102] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8371.791510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8371.804959] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8372.957652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8372.973243] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8373.012046] Lustre: lustre-MDT0000: new disk, initializing [ 8373.070710] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8373.080365] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8374.700715] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8380.017150] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8380.084081] Lustre: 473340:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8380.111870] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8380.114337] Lustre: Skipped 1 previous similar message [ 8380.153783] Lustre: lustre-MDT0001: new disk, initializing [ 8380.204530] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8380.224306] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8380.229867] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8382.134045] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8385.978566] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8387.425778] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8390.903986] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8390.996279] Lustre: *** cfs_fail_loc=90f, val=0*** [ 8391.030921] Lustre: lustre-OST0000: new disk, initializing [ 8391.033610] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8391.038281] Lustre: 475130:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8391.081387] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8393.564232] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8397.411260] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8397.831827] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8397.838421] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8397.902498] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8402.915891] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8402.919429] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8402.920905] LustreError: Skipped 1 previous similar message [ 8402.931911] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8407.020630] Lustre: server umount lustre-OST0000 complete [ 8411.106730] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8411.111728] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8411.120267] Lustre: Skipped 1 previous similar message [ 8411.123814] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8411.126858] Lustre: Skipped 1 previous similar message [ 8413.153846] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8413.154251] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8413.162690] Lustre: Skipped 1 previous similar message [ 8418.275752] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8418.279500] Lustre: Skipped 1 previous similar message [ 8423.392159] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8423.393555] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8423.401047] Lustre: Skipped 1 previous similar message [ 8423.475830] Lustre: server umount lustre-MDT0000 complete [ 8425.170937] LustreError: 474119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781226943 with bad export cookie 3662581655376491032 [ 8425.174640] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8425.175225] LustreError: 474119:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8425.322407] Lustre: server umount lustre-MDT0001 complete [ 8429.770155] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 8430.971262] Key type lgssc unregistered [ 8431.124436] LNet: 476901:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8431.128258] LNetError: 476901:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8431.137459] LNet: Removed LNI 192.168.206.140@tcp [ 8431.491312] Key type .llcrypt unregistered [ 8431.493074] Key type ._llcrypt unregistered [ 8440.535319] Key type ._llcrypt registered [ 8440.536860] Key type .llcrypt registered [ 8440.585840] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8441.053536] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8441.094706] alg: No test for adler32 (adler32-zlib) [ 8441.985545] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 8442.088782] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 8443.680304] Key type lgssc registered [ 8444.163843] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8447.840512] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8447.853369] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8449.074937] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8450.825251] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8455.242759] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8455.442197] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8457.172510] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8459.525994] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8461.168577] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8464.339437] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8464.510210] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8467.122548] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8469.533153] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8480.196346] loop0: detected capacity change from 0 to 204800000 [ 8480.273490] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8480.278680] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8480.284985] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8483.029463] loop0: detected capacity change from 0 to 204800000 [ 8483.149024] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8483.154901] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8483.163604] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 8485.350293] loop0: detected capacity change from 0 to 204800000 [ 8487.792431] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8487.865781] Lustre: fs15246-OST0000: new disk, initializing [ 8487.868951] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [ 8487.872677] Lustre: 481381:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8487.904778] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8490.391380] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8494.964514] loop1: detected capacity change from 0 to 204800000 [ 8497.214932] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8497.264701] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [ 8497.290439] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [ 8497.332383] Lustre: fs15246-MDT0000: new disk, initializing [ 8497.399505] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8497.412865] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8497.471755] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 8498.529407] Lustre: 482485:0:(client.c:1638:after_reply()) @@@ resending request on EINPROGRESS req@00000000539a01b4 x1867751840842368/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1781227032 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 8498.994979] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8507.619228] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [ 8507.625352] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [ 8507.636269] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [ 8508.361082] Lustre: server umount fs15246-MDT0000 complete [ 8509.924239] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8509.932071] LustreError: 479280:0:(ldlm_lib.c:1179:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8509.944368] LustreError: 479280:0:(ldlm_lib.c:1179:target_handle_connect()) Skipped 2 previous similar messages [ 8514.445378] Lustre: server umount fs15246-OST0000 complete [ 8520.164063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8520.164687] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8520.170806] Lustre: Skipped 1 previous similar message [ 8525.284601] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8525.289140] Lustre: Skipped 2 previous similar messages [ 8525.469488] Lustre: server umount lustre-OST0000 complete [ 8527.329034] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8527.334116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8527.340211] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8530.402617] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8530.402668] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8530.412537] Lustre: Skipped 1 previous similar message [ 8533.282152] Lustre: server umount lustre-MDT0000 complete [ 8534.823570] LustreError: 478110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227052 with bad export cookie 5502583996832106352 [ 8534.824150] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8534.831636] LustreError: 478110:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8534.973115] Lustre: server umount lustre-MDT0001 complete [ 8539.465460] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 8540.698953] Key type lgssc unregistered [ 8540.841587] LNet: 485542:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8540.846442] LNetError: 485542:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8540.867486] LNet: Removed LNI 192.168.206.140@tcp [ 8541.203092] Key type .llcrypt unregistered [ 8541.204230] Key type ._llcrypt unregistered [ 8550.504634] Key type ._llcrypt registered [ 8550.506025] Key type .llcrypt registered [ 8550.552041] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8550.969881] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8551.038771] alg: No test for adler32 (adler32-zlib) [ 8551.927204] Lustre: Lustre: Build Version: 2.17.53_74_g6bc6b37 [ 8552.041380] LNet: Added LNI 192.168.206.140@tcp [8/256/0/180] [ 8553.632468] Key type lgssc registered [ 8554.117254] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8571.992867] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 8576.584856] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8591.765241] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8591.776875] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8592.894619] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8592.914483] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8592.954577] Lustre: lustre-MDT0000: new disk, initializing [ 8592.996683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8593.007681] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8594.704835] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8599.926932] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8599.982677] Lustre: 489929:0:(mgs_llog.c:1437:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8600.011663] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8600.014745] Lustre: Skipped 1 previous similar message [ 8600.055108] Lustre: lustre-MDT0001: new disk, initializing [ 8600.096222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8600.117068] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8600.127642] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8601.911428] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8606.075852] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid [ 8607.676415] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid [ 8610.582366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8610.699789] Lustre: lustre-OST0000: new disk, initializing [ 8610.702606] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8610.706700] Lustre: 491670:0:(osd_compat.c:1352:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8610.749866] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8610.849445] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8610.854107] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8610.915801] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8613.149202] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_default_debug -1 all [ 8616.999210] Lustre: DEBUG MARKER: oleg640-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid [ 8618.749654] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8618.838927] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8620.611773] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8620.719309] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8626.153152] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8626.153244] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8626.155655] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8626.159296] Lustre: Skipped 1 previous similar message [ 8626.162296] LustreError: Skipped 1 previous similar message [ 8627.680965] Lustre: server umount lustre-OST0000 complete [ 8630.754153] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8630.758338] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8630.764516] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8630.767089] Lustre: Skipped 1 previous similar message [ 8635.383877] Lustre: server umount lustre-MDT0000 complete [ 8636.897924] LustreError: 491669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1781227155 with bad export cookie 16359111317827104210 [ 8636.902505] LustreError: 491669:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8636.902828] LustreError: MGC192.168.206.140@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8637.039867] Lustre: server umount lustre-MDT0001 complete [ 8647.666131] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing set_hostid [ 8652.448112] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing load_modules_local [ 8666.558102] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 8514 sec ========== 21:19:44 (1781227184) [ 8667.356837] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 21:19:45 (1781227185) === [ 8668.846274] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 21:19:47 (1781227187) === [ 8687.273794] Lustre: DEBUG MARKER: oleg640-server.virtnet: executing unload_modules_local [ 8688.565093] Key type lgssc unregistered [ 8688.713474] LNet: 498102:0:(lib-ptl.c:967:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8688.716660] LNetError: 498102:0:(acceptor.c:246:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8688.727483] LNet: Removed LNI 192.168.206.140@tcp [ 8689.112500] Key type .llcrypt unregistered [ 8689.114141] Key type ._llcrypt unregistered