[ 0.000000] Linux version 4.18.0rh8.10-debug (green@maintenance) (gcc version 8.5.0 20210514 (Red Hat 8.5.0-26) (GCC)) #2 SMP Mon Jul 14 01:24:22 EDT 2025 [ 0.000000] Command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] x86/fpu: Supporting XSAVE feature 0x001: 'x87 floating point registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x002: 'SSE registers' [ 0.000000] x86/fpu: Supporting XSAVE feature 0x004: 'AVX registers' [ 0.000000] x86/fpu: xstate_offset[2]: 576, xstate_sizes[2]: 256 [ 0.000000] x86/fpu: Enabled xstate features 0x7, context size is 832 bytes, using 'standard' format. [ 0.000000] signal: max sigframe size: 1776 [ 0.000000] BIOS-provided physical RAM map: [ 0.000000] BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable [ 0.000000] BIOS-e820: [mem 0x000000000009fc00-0x000000000009ffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000000f0000-0x00000000000fffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000000100000-0x00000000bffcdfff] usable [ 0.000000] BIOS-e820: [mem 0x00000000bffce000-0x00000000bfffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000feffc000-0x00000000feffffff] reserved [ 0.000000] BIOS-e820: [mem 0x00000000fffc0000-0x00000000ffffffff] reserved [ 0.000000] BIOS-e820: [mem 0x0000000100000000-0x0000000146dfffff] usable [ 0.000000] NX (Execute Disable) protection: active [ 0.000000] SMBIOS 2.8 present. [ 0.000000] DMI: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.17.0-10.fc44 06/10/2025 [ 0.000000] Hypervisor detected: KVM [ 0.000000] kvm-clock: Using msrs 4b564d01 and 4b564d00 [ 0.000000] kvm-clock: using sched offset of 437084940 cycles [ 0.000000] clocksource: kvm-clock: mask: 0xffffffffffffffff max_cycles: 0x1cd42e4dffb, max_idle_ns: 881590591483 ns [ 0.000000] tsc: Detected 2399.998 MHz processor [ 0.000000] last_pfn = 0x146e00 max_arch_pfn = 0x400000000 [ 0.000000] x86/PAT: Configuration [0-7]: WB WC UC- UC WB WP UC- WT [ 0.000000] last_pfn = 0xbffce max_arch_pfn = 0x400000000 [ 0.000000] found SMP MP-table at [mem 0x000f54b0-0x000f54bf] [ 0.000000] RAMDISK: [mem 0xbcc54000-0xbffbffff] [ 0.000000] ACPI: Early table checksum verification disabled [ 0.000000] ACPI: RSDP 0x00000000000F52D0 000014 (v00 BOCHS ) [ 0.000000] ACPI: RSDT 0x00000000BFFE247C 000034 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACP 0x00000000BFFE2318 000074 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: DSDT 0x00000000BFFE0040 0022D8 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: FACS 0x00000000BFFE0000 000040 [ 0.000000] ACPI: APIC 0x00000000BFFE238C 000090 (v03 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: HPET 0x00000000BFFE241C 000038 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: WAET 0x00000000BFFE2454 000028 (v01 BOCHS BXPC 00000001 BXPC 00000001) [ 0.000000] ACPI: Reserving FACP table memory at [mem 0xbffe2318-0xbffe238b] [ 0.000000] ACPI: Reserving DSDT table memory at [mem 0xbffe0040-0xbffe2317] [ 0.000000] ACPI: Reserving FACS table memory at [mem 0xbffe0000-0xbffe003f] [ 0.000000] ACPI: Reserving APIC table memory at [mem 0xbffe238c-0xbffe241b] [ 0.000000] ACPI: Reserving HPET table memory at [mem 0xbffe241c-0xbffe2453] [ 0.000000] ACPI: Reserving WAET table memory at [mem 0xbffe2454-0xbffe247b] [ 0.000000] No NUMA configuration found [ 0.000000] Faking a node at [mem 0x0000000000000000-0x0000000146dfffff] [ 0.000000] NODE_DATA(0) allocated [mem 0x1465a3000-0x1465cdfff] [ 0.000000] Reserving 256MB of memory at 2752MB for crashkernel (System RAM: 4205MB) [ 0.000000] Zone ranges: [ 0.000000] DMA [mem 0x0000000000001000-0x0000000000ffffff] [ 0.000000] DMA32 [mem 0x0000000001000000-0x00000000ffffffff] [ 0.000000] Normal [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Device empty [ 0.000000] Movable zone start for each node [ 0.000000] Early memory node ranges [ 0.000000] node 0: [mem 0x0000000000001000-0x000000000009efff] [ 0.000000] node 0: [mem 0x0000000000100000-0x00000000bffcdfff] [ 0.000000] node 0: [mem 0x0000000100000000-0x0000000146dfffff] [ 0.000000] Zeroed struct page in unavailable ranges: 4756 pages [ 0.000000] Initmem setup node 0 [mem 0x0000000000001000-0x0000000146dfffff] [ 0.000000] ACPI: PM-Timer IO Port: 0x608 [ 0.000000] ACPI: LAPIC_NMI (acpi_id[0xff] dfl dfl lint[0x1]) [ 0.000000] IOAPIC[0]: apic_id 0, version 17, address 0xfec00000, GSI 0-23 [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 0 global_irq 2 dfl dfl) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 5 global_irq 5 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 9 global_irq 9 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 10 global_irq 10 high level) [ 0.000000] ACPI: INT_SRC_OVR (bus 0 bus_irq 11 global_irq 11 high level) [ 0.000000] Using ACPI (MADT) for SMP configuration information [ 0.000000] ACPI: HPET id: 0x8086a201 base: 0xfed00000 [ 0.000000] TSC deadline timer available [ 0.000000] smpboot: Allowing 4 CPUs, 0 hotplug CPUs [ 0.000000] kvm-guest: KVM setup pv remote TLB flush [ 0.000000] kvm-guest: setup PV sched yield [ 0.000000] PM: Registered nosave memory: [mem 0x00000000-0x00000fff] [ 0.000000] PM: Registered nosave memory: [mem 0x0009f000-0x0009ffff] [ 0.000000] PM: Registered nosave memory: [mem 0x000a0000-0x000effff] [ 0.000000] PM: Registered nosave memory: [mem 0x000f0000-0x000fffff] [ 0.000000] PM: Registered nosave memory: [mem 0xbffce000-0xbfffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xc0000000-0xfeffbfff] [ 0.000000] PM: Registered nosave memory: [mem 0xfeffc000-0xfeffffff] [ 0.000000] PM: Registered nosave memory: [mem 0xff000000-0xfffbffff] [ 0.000000] PM: Registered nosave memory: [mem 0xfffc0000-0xffffffff] [ 0.000000] [mem 0xc0000000-0xfeffbfff] available for PCI devices [ 0.000000] Booting paravirtualized kernel on KVM [ 0.000000] clocksource: refined-jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1910969940391419 ns [ 0.000000] setup_percpu: NR_CPUS:8192 nr_cpumask_bits:4 nr_cpu_ids:4 nr_node_ids:1 [ 0.000000] percpu: Embedded 63 pages/cpu s221184 r8192 d28672 u524288 [ 0.000000] kvm-guest: PV spinlocks enabled [ 0.000000] PV qspinlock hash table entries: 256 (order: 0, 4096 bytes, linear) [ 0.000000] Built 1 zonelists, mobility grouping on. Total pages: 1059606 [ 0.000000] Policy zone: Normal [ 0.000000] Kernel command line: rd.shell root=nbd:192.168.200.253:rocky8.10:ext4:ro:-p,-b4096 ro crashkernel=256M panic=1 nomodeset ipmtu=9000 ip=dhcp rd.neednet=1 init_on_free=off mitigations=off console=ttyS1,115200 audit=0 [ 0.000000] Specific versions of hardware are certified with Red Hat Enterprise Linux 8. Please see the list of hardware certified with Red Hat Enterprise Linux 8 at https://catalog.redhat.com. [ 0.000000] audit: disabled (until reboot) [ 0.000000] software IO TLB: area num 4. [ 0.000000] Memory: 2895240K/4306352K available (18435K kernel code, 11221K rwdata, 7248K rodata, 2908K init, 18040K bss, 524584K reserved, 0K cma-reserved) [ 0.000000] SLUB: HWalign=64, Order=0-3, MinObjects=0, CPUs=4, Nodes=1 [ 0.000000] kmemleak: Kernel memory leak detector disabled [ 0.000000] ftrace: allocating 41240 entries in 162 pages [ 0.000000] ftrace: allocated 162 pages with 3 groups [ 0.000000] rcu: Hierarchical RCU implementation. [ 0.000000] rcu: RCU event tracing is enabled. [ 0.000000] rcu: RCU restricting CPUs from NR_CPUS=8192 to nr_cpu_ids=4. [ 0.000000] rcu: RCU callback double-/use-after-free debug enabled. [ 0.000000] Rude variant of Tasks RCU enabled. [ 0.000000] Tracing variant of Tasks RCU enabled. [ 0.000000] rcu: RCU calculated value of scheduler-enlistment delay is 100 jiffies. [ 0.000000] rcu: Adjusting geometry for rcu_fanout_leaf=16, nr_cpu_ids=4 [ 0.000000] NR_IRQS: 524544, nr_irqs: 456, preallocated irqs: 16 [ 0.000000] random: get_random_bytes called from start_kernel+0x622/0x9a8 with crng_init=0 [ 0.001000] Console: colour *CGA 80x25 [ 0.001000] printk: console [ttyS1] enabled [ 0.001000] ACPI: Core revision 20220331 [ 0.001000] clocksource: hpet: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 19112604467 ns [ 0.001012] APIC: Switch to symmetric I/O mode setup [ 0.002246] x2apic enabled [ 0.004004] Switched APIC routing to physical x2apic. [ 0.005012] kvm-guest: setup PV IPIs [ 0.008000] ..TIMER: vector=0x30 apic1=0 pin1=2 apic2=-1 pin2=-1 [ 0.008000] clocksource: tsc-early: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 0.008017] Calibrating delay loop (skipped) preset value.. 4799.99 BogoMIPS (lpj=2399998) [ 0.009011] pid_max: default: 32768 minimum: 301 [ 0.011102] LSM: Security Framework initializing [ 0.012040] Yama: becoming mindful. [ 0.013028] SELinux: Initializing. [ 0.014058] *** VALIDATE selinux *** [ 0.021642] Dentry cache hash table entries: 1048576 (order: 11, 8388608 bytes, vmalloc) [ 0.025244] Inode-cache hash table entries: 524288 (order: 10, 4194304 bytes, vmalloc) [ 0.026148] Mount-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.028076] Mountpoint-cache hash table entries: 16384 (order: 5, 131072 bytes, vmalloc) [ 0.029103] *** VALIDATE tmpfs *** [ 0.030393] *** VALIDATE proc *** [ 0.031143] *** VALIDATE cgroup *** [ 0.032006] *** VALIDATE cgroup2 *** [ 0.033261] x86/cpu: User Mode Instruction Prevention (UMIP) activated [ 0.034106] Last level iTLB entries: 4KB 0, 2MB 0, 4MB 0 [ 0.035004] Last level dTLB entries: 4KB 0, 2MB 0, 4MB 0, 1GB 0 [ 0.036026] Spectre V2 : User space: Vulnerable [ 0.037007] Speculative Store Bypass: Vulnerable [ 0.040246] debug: unmapping init [mem 0xffffffff97059000-0xffffffff97060fff] [ 0.042857] smpboot: CPU0: Intel(R) Xeon(R) CPU E5-2695 v2 @ 2.40GHz (family: 0x6, model: 0x3e, stepping: 0x4) [ 0.043670] Performance Events: IvyBridge events, full-width counters, Intel PMU driver. [ 0.044022] ... version: 2 [ 0.045013] ... bit width: 48 [ 0.045896] ... generic registers: 4 [ 0.046014] ... value mask: 0000ffffffffffff [ 0.047008] ... max period: 00007fffffffffff [ 0.048008] ... fixed-purpose events: 3 [ 0.049007] ... event mask: 000000070000000f [ 0.050249] rcu: Hierarchical SRCU implementation. [ 0.052260] smp: Bringing up secondary CPUs ... [ 0.053503] x86: Booting SMP configuration: [ 0.054021] .... node #0, CPUs: #1 #2 #3 [ 0.059024] smp: Brought up 1 node, 4 CPUs [ 0.061013] smpboot: Max logical packages: 1 [ 0.062015] smpboot: Total of 4 processors activated (19199.98 BogoMIPS) [ 0.247303] node 0 deferred pages initialised in 183ms [ 0.250157] devtmpfs: initialized [ 0.251214] x86/mm: Memory block size: 128MB [ 0.253694] gcov: version magic: 0x41383552 [ 0.255143] clocksource: jiffies: mask: 0xffffffff max_cycles: 0xffffffff, max_idle_ns: 1911260446275000 ns [ 0.256163] futex hash table entries: 1024 (order: 4, 65536 bytes, vmalloc) [ 0.258441] pinctrl core: initialized pinctrl subsystem [ 0.260202] [ 0.260532] ************************************************************* [ 0.262014] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.263007] ** ** [ 0.265010] ** IOMMU DebugFS SUPPORT HAS BEEN ENABLED IN THIS KERNEL ** [ 0.267014] ** ** [ 0.268025] ** This means that this kernel is built to expose internal ** [ 0.270011] ** IOMMU data structures, which may compromise security on ** [ 0.272017] ** your system. ** [ 0.274014] ** ** [ 0.276013] ** If you see this message and you are not debugging the ** [ 0.278013] ** kernel, report this immediately to your vendor! ** [ 0.280014] ** ** [ 0.282010] ** NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE NOTICE ** [ 0.284019] ************************************************************* [ 0.287766] NET: Registered protocol family 16 [ 0.289492] DMA: preallocated 512 KiB GFP_KERNEL pool for atomic allocations [ 0.292076] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA pool for atomic allocations [ 0.295117] DMA: preallocated 512 KiB GFP_KERNEL|GFP_DMA32 pool for atomic allocations [ 0.299050] cpuidle: using governor menu [ 0.300681] acpiphp: ACPI Hot Plug PCI Controller Driver version: 0.5 [ 0.302349] PCI: Using configuration type 1 for base access [ 0.305134] core: PMU erratum BJ122, BV98, HSD29 worked around, HT is on [ 0.313065] HugeTLB registered 1.00 GiB page size, pre-allocated 0 pages [ 0.314025] HugeTLB registered 2.00 MiB page size, pre-allocated 0 pages [ 0.316033] cryptd: max_cpu_qlen set to 1000 [ 0.318221] ACPI: Added _OSI(Module Device) [ 0.320017] ACPI: Added _OSI(Processor Device) [ 0.322011] ACPI: Added _OSI(3.0 _SCP Extensions) [ 0.324015] ACPI: Added _OSI(Processor Aggregator Device) [ 0.329455] ACPI: 1 ACPI AML tables successfully acquired and loaded [ 0.336122] ACPI: Interpreter enabled [ 0.337047] ACPI: PM: (supports S0 S3 S4 S5) [ 0.338022] ACPI: Using IOAPIC for interrupt routing [ 0.340082] PCI: Using host bridge windows from ACPI; if necessary, use "pci=nocrs" and report a bug [ 0.342393] ACPI: Enabled 2 GPEs in block 00 to 0F [ 0.352428] ACPI: PCI Root Bridge [PCI0] (domain 0000 [bus 00-ff]) [ 0.353000] acpi PNP0A03:00: _OSC: OS supports [ASPM ClockPM Segments MSI HPX-Type3] [ 0.357023] acpi PNP0A03:00: _OSC: not requesting OS control; OS requires [ExtendedConfig ASPM ClockPM MSI] [ 0.359079] acpi PNP0A03:00: fail to add MMCONFIG information, can't access extended PCI configuration space under this bridge. [ 0.363430] acpiphp: Slot [2] registered [ 0.365154] acpiphp: Slot [5] registered [ 0.366112] acpiphp: Slot [6] registered [ 0.367104] acpiphp: Slot [7] registered [ 0.369130] acpiphp: Slot [8] registered [ 0.370137] acpiphp: Slot [9] registered [ 0.372166] acpiphp: Slot [10] registered [ 0.373118] acpiphp: Slot [3] registered [ 0.374066] acpiphp: Slot [4] registered [ 0.375091] acpiphp: Slot [11] registered [ 0.377080] acpiphp: Slot [12] registered [ 0.378094] acpiphp: Slot [13] registered [ 0.379097] acpiphp: Slot [14] registered [ 0.380090] acpiphp: Slot [15] registered [ 0.382082] acpiphp: Slot [16] registered [ 0.382902] acpiphp: Slot [17] registered [ 0.383055] acpiphp: Slot [18] registered [ 0.384116] acpiphp: Slot [19] registered [ 0.386096] acpiphp: Slot [20] registered [ 0.388113] acpiphp: Slot [21] registered [ 0.389121] acpiphp: Slot [22] registered [ 0.391085] acpiphp: Slot [23] registered [ 0.392101] acpiphp: Slot [24] registered [ 0.394074] acpiphp: Slot [25] registered [ 0.395074] acpiphp: Slot [26] registered [ 0.397092] acpiphp: Slot [27] registered [ 0.398057] acpiphp: Slot [28] registered [ 0.399112] acpiphp: Slot [29] registered [ 0.401184] acpiphp: Slot [30] registered [ 0.403127] acpiphp: Slot [31] registered [ 0.404062] PCI host bridge to bus 0000:00 [ 0.406016] pci_bus 0000:00: root bus resource [io 0x0000-0x0cf7 window] [ 0.408024] pci_bus 0000:00: root bus resource [io 0x0d00-0xffff window] [ 0.411023] pci_bus 0000:00: root bus resource [mem 0x000a0000-0x000bffff window] [ 0.414035] pci_bus 0000:00: root bus resource [mem 0xc0000000-0xfebfffff window] [ 0.415030] pci_bus 0000:00: root bus resource [mem 0xe0000000000-0xe007fffffff window] [ 0.418036] pci_bus 0000:00: root bus resource [bus 00-ff] [ 0.420190] pci 0000:00:00.0: [8086:1237] type 00 class 0x060000 [ 0.423240] pci 0000:00:01.0: [8086:7000] type 00 class 0x060100 [ 0.426407] pci 0000:00:01.1: [8086:7010] type 00 class 0x010180 [ 0.435783] pci 0000:00:01.1: reg 0x20: [io 0xc320-0xc32f] [ 0.440583] pci 0000:00:01.1: legacy IDE quirk: reg 0x10: [io 0x01f0-0x01f7] [ 0.443025] pci 0000:00:01.1: legacy IDE quirk: reg 0x14: [io 0x03f6] [ 0.446038] pci 0000:00:01.1: legacy IDE quirk: reg 0x18: [io 0x0170-0x0177] [ 0.448024] pci 0000:00:01.1: legacy IDE quirk: reg 0x1c: [io 0x0376] [ 0.452555] pci 0000:00:01.3: [8086:7113] type 00 class 0x068000 [ 0.455892] pci 0000:00:01.3: quirk: [io 0x0600-0x063f] claimed by PIIX4 ACPI [ 0.458049] pci 0000:00:01.3: quirk: [io 0x0700-0x070f] claimed by PIIX4 SMB [ 0.460923] pci 0000:00:02.0: [1af4:1000] type 00 class 0x020000 [ 0.467020] pci 0000:00:02.0: reg 0x10: [io 0xc300-0xc31f] [ 0.478886] pci 0000:00:02.0: reg 0x20: [mem 0xe0000000000-0xe0000003fff 64bit pref] [ 0.486016] pci 0000:00:02.0: reg 0x30: [mem 0xfeb80000-0xfebbffff pref] [ 0.492387] pci 0000:00:05.0: [1af4:1001] type 00 class 0x010000 [ 0.499022] pci 0000:00:05.0: reg 0x10: [io 0xc000-0xc07f] [ 0.504020] pci 0000:00:05.0: reg 0x14: [mem 0xfebc0000-0xfebc0fff] [ 0.517018] pci 0000:00:05.0: reg 0x20: [mem 0xe0000004000-0xe0000007fff 64bit pref] [ 0.528154] pci 0000:00:06.0: [1af4:1001] type 00 class 0x010000 [ 0.533013] pci 0000:00:06.0: reg 0x10: [io 0xc080-0xc0ff] [ 0.537012] pci 0000:00:06.0: reg 0x14: [mem 0xfebc1000-0xfebc1fff] [ 0.551016] pci 0000:00:06.0: reg 0x20: [mem 0xe0000008000-0xe000000bfff 64bit pref] [ 0.561386] pci 0000:00:07.0: [1af4:1001] type 00 class 0x010000 [ 0.568019] pci 0000:00:07.0: reg 0x10: [io 0xc100-0xc17f] [ 0.573018] pci 0000:00:07.0: reg 0x14: [mem 0xfebc2000-0xfebc2fff] [ 0.589030] pci 0000:00:07.0: reg 0x20: [mem 0xe000000c000-0xe000000ffff 64bit pref] [ 0.600294] pci 0000:00:08.0: [1af4:1001] type 00 class 0x010000 [ 0.606017] pci 0000:00:08.0: reg 0x10: [io 0xc180-0xc1ff] [ 0.614019] pci 0000:00:08.0: reg 0x14: [mem 0xfebc3000-0xfebc3fff] [ 0.637016] pci 0000:00:08.0: reg 0x20: [mem 0xe0000010000-0xe0000013fff 64bit pref] [ 0.647419] pci 0000:00:09.0: [1af4:1001] type 00 class 0x010000 [ 0.652881] pci 0000:00:09.0: reg 0x10: [io 0xc200-0xc27f] [ 0.661018] pci 0000:00:09.0: reg 0x14: [mem 0xfebc4000-0xfebc4fff] [ 0.678020] pci 0000:00:09.0: reg 0x20: [mem 0xe0000014000-0xe0000017fff 64bit pref] [ 0.686000] pci 0000:00:0a.0: [1af4:1001] type 00 class 0x010000 [ 0.692017] pci 0000:00:0a.0: reg 0x10: [io 0xc280-0xc2ff] [ 0.697019] pci 0000:00:0a.0: reg 0x14: [mem 0xfebc5000-0xfebc5fff] [ 0.716015] pci 0000:00:0a.0: reg 0x20: [mem 0xe0000018000-0xe000001bfff 64bit pref] [ 0.725916] ACPI: PCI: Interrupt link LNKA configured for IRQ 10 [ 0.727423] ACPI: PCI: Interrupt link LNKB configured for IRQ 10 [ 0.730392] ACPI: PCI: Interrupt link LNKC configured for IRQ 11 [ 0.734417] ACPI: PCI: Interrupt link LNKD configured for IRQ 11 [ 0.736192] ACPI: PCI: Interrupt link LNKS configured for IRQ 9 [ 0.740205] iommu: Default domain type: Passthrough [ 0.741653] SCSI subsystem initialized [ 0.744196] ACPI: bus type USB registered [ 0.746149] usbcore: registered new interface driver usbfs [ 0.748108] usbcore: registered new interface driver hub [ 0.750101] usbcore: registered new device driver usb [ 0.751176] pps_core: LinuxPPS API ver. 1 registered [ 0.753016] pps_core: Software ver. 5.3.6 - Copyright 2005-2007 Rodolfo Giometti [ 0.756053] PTP clock support registered [ 0.758191] EDAC MC: Ver: 3.0.0 [ 0.760151] PCI: Using ACPI for IRQ routing [ 0.761893] NetLabel: Initializing [ 0.763009] NetLabel: domain hash size = 128 [ 0.764009] NetLabel: protocols = UNLABELED CIPSOv4 CALIPSO [ 0.766094] NetLabel: unlabeled traffic allowed by default [ 0.768294] vgaarb: loaded [ 0.769293] hpet0: at MMIO 0xfed00000, IRQs 2, 8, 0 [ 0.770013] hpet0: 3 comparators, 64-bit 100.000000 MHz counter [ 0.775000] clocksource: Switched to clocksource kvm-clock [ 0.876526] VFS: Disk quotas dquot_6.6.0 [ 0.877926] VFS: Dquot-cache hash table entries: 512 (order 0, 4096 bytes) [ 0.880137] *** VALIDATE ramfs *** [ 0.881079] *** VALIDATE hugetlbfs *** [ 0.882322] pnp: PnP ACPI init [ 0.885083] pnp: PnP ACPI: found 6 devices [ 0.902796] clocksource: acpi_pm: mask: 0xffffff max_cycles: 0xffffff, max_idle_ns: 2085701024 ns [ 0.905826] pci_bus 0000:00: resource 4 [io 0x0000-0x0cf7 window] [ 0.908169] pci_bus 0000:00: resource 5 [io 0x0d00-0xffff window] [ 0.910049] pci_bus 0000:00: resource 6 [mem 0x000a0000-0x000bffff window] [ 0.912319] pci_bus 0000:00: resource 7 [mem 0xc0000000-0xfebfffff window] [ 0.914943] pci_bus 0000:00: resource 8 [mem 0xe0000000000-0xe007fffffff window] [ 0.917864] NET: Registered protocol family 2 [ 0.920347] IP idents hash table entries: 131072 (order: 8, 1048576 bytes, vmalloc) [ 0.925534] tcp_listen_portaddr_hash hash table entries: 4096 (order: 5, 163840 bytes, vmalloc) [ 0.929070] TCP established hash table entries: 65536 (order: 7, 524288 bytes, vmalloc) [ 0.934344] TCP bind hash table entries: 65536 (order: 9, 2097152 bytes, vmalloc) [ 0.937862] TCP: Hash tables configured (established 65536 bind 65536) [ 0.940152] MPTCP token hash table entries: 8192 (order: 6, 393216 bytes, vmalloc) [ 0.942136] UDP hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.944372] UDP-Lite hash table entries: 4096 (order: 6, 393216 bytes, vmalloc) [ 0.946869] NET: Registered protocol family 1 [ 0.948628] RPC: Registered named UNIX socket transport module. [ 0.950154] RPC: Registered udp transport module. [ 0.951537] RPC: Registered tcp transport module. [ 0.953043] RPC: Registered tcp NFSv4.1 backchannel transport module. [ 0.955113] NET: Registered protocol family 44 [ 0.957134] pci 0000:00:00.0: Limiting direct PCI/PCI transfers [ 0.959474] pci 0000:00:01.0: PIIX3: Enabling Passive Release [ 0.962073] pci 0000:00:01.0: Activating ISA DMA hang workarounds [ 0.964835] PCI: CLS 0 bytes, default 64 [ 0.967068] Unpacking initramfs... [ 2.343701] debug: unmapping init [mem 0xffff97e23cc54000-0xffff97e23ffbffff] [ 2.346725] PCI-DMA: Using software bounce buffering for IO (SWIOTLB) [ 2.348250] software IO TLB: mapped [mem 0x00000000a8000000-0x00000000ac000000] (64MB) [ 2.350251] clocksource: tsc: mask: 0xffffffffffffffff max_cycles: 0x229835b7123, max_idle_ns: 440795242976 ns [ 2.822074] Initialise system trusted keyrings [ 2.823269] Key type blacklist registered [ 2.825728] workingset: timestamp_bits=36 max_order=20 bucket_order=0 [ 2.832262] zbud: loaded [ 2.834940] *** VALIDATE nfs *** [ 2.836282] *** VALIDATE nfs4 *** [ 2.838032] pstore: using deflate compression [ 2.840747] Platform Keyring initialized [ 2.942327] NET: Registered protocol family 38 [ 2.945063] Key type asymmetric registered [ 2.946822] Asymmetric key parser 'x509' registered [ 2.951212] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 247) [ 2.954633] io scheduler mq-deadline registered [ 2.956646] io scheduler kyber registered [ 2.958755] io scheduler bfq registered [ 2.964407] atomic64_test: passed for x86-64 platform with CX8 and with SSE [ 2.967316] shpchp: Standard Hot Plug PCI Controller Driver version: 0.4 [ 2.970422] input: Power Button as /devices/LNXSYSTM:00/LNXPWRBN:00/input/input0 [ 2.973191] ACPI: Power Button [PWRF] [ 2.979668] ACPI: \_SB_.LNKB: Enabled at IRQ 10 [ 2.986732] ACPI: \_SB_.LNKA: Enabled at IRQ 11 [ 2.997351] ACPI: \_SB_.LNKC: Enabled at IRQ 11 [ 3.003688] ACPI: \_SB_.LNKD: Enabled at IRQ 10 [ 3.021646] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled [ 3.052626] 00:03: ttyS1 at I/O 0x2f8 (irq = 3, base_baud = 115200) is a 16550A [ 3.081608] 00:04: ttyS0 at I/O 0x3f8 (irq = 4, base_baud = 115200) is a 16550A [ 3.087669] Non-volatile memory driver v1.3 [ 3.089347] Linux agpgart interface v0.103 [ 3.128054] virtio_blk virtio1: [vda] 146232 512-byte logical blocks (74.9 MB/71.4 MiB) [ 3.131149] vda: detected capacity change from 0 to 74870784 [ 3.145158] virtio_blk virtio2: [vdb] 2097152 512-byte logical blocks (1.07 GB/1.00 GiB) [ 3.148411] vdb: detected capacity change from 0 to 1073741824 [ 3.165094] virtio_blk virtio3: [vdc] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.168040] vdc: detected capacity change from 0 to 2621440000 [ 3.185882] virtio_blk virtio4: [vdd] 5120000 512-byte logical blocks (2.62 GB/2.44 GiB) [ 3.188989] vdd: detected capacity change from 0 to 2621440000 [ 3.202059] virtio_blk virtio5: [vde] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.205361] vde: detected capacity change from 0 to 4294967296 [ 3.220269] virtio_blk virtio6: [vdf] 8388608 512-byte logical blocks (4.29 GB/4.00 GiB) [ 3.223592] vdf: detected capacity change from 0 to 4294967296 [ 3.230623] libphy: Fixed MDIO Bus: probed [ 3.237588] usbcore: registered new interface driver usbserial_generic [ 3.240350] usbserial: USB Serial support registered for generic [ 3.242810] i8042: PNP: PS/2 Controller [PNP0303:KBD,PNP0f13:MOU] at 0x60,0x64 irq 1,12 [ 3.246780] serio: i8042 KBD port at 0x60,0x64 irq 1 [ 3.248556] serio: i8042 AUX port at 0x60,0x64 irq 12 [ 3.251414] mousedev: PS/2 mouse device common for all mice [ 3.254758] rtc_cmos 00:05: RTC can wake from S4 [ 3.257367] input: AT Translated Set 2 keyboard as /devices/platform/i8042/serio0/input/input1 [ 3.257698] rtc_cmos 00:05: registered as rtc0 [ 3.261995] rtc_cmos 00:05: alarms up to one day, y3k, 242 bytes nvram, hpet irqs [ 3.262678] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input4 [ 3.264357] intel_pstate: CPU model not supported [ 3.267706] input: VirtualPS/2 VMware VMMouse as /devices/platform/i8042/serio1/input/input3 [ 3.271460] hid: raw HID events driver (C) Jiri Kosina [ 3.273178] usbcore: registered new interface driver usbhid [ 3.274803] usbhid: USB HID core driver [ 3.276111] drop_monitor: Initializing network drop monitor service [ 3.278755] Initializing XFRM netlink socket [ 3.280985] NET: Registered protocol family 10 [ 3.284095] Segment Routing with IPv6 [ 3.285741] NET: Registered protocol family 17 [ 3.288380] mpls_gso: MPLS GSO support [ 3.295224] RAS: Correctable Errors collector initialized. [ 3.297542] AVX version of gcm_enc/dec engaged. [ 3.299366] AES CTR mode by8 optimization enabled [ 3.375411] sched_clock: Marking stable (3375389787, 0)->(4214234974, -838845187) [ 3.379320] registered taskstats version 1 [ 3.381370] Loading compiled-in X.509 certificates [ 3.383548] zswap: loaded using pool lzo/zbud [ 3.406483] Key type big_key registered [ 3.428751] Key type encrypted registered [ 3.430495] ima: No TPM chip found, activating TPM-bypass! [ 3.432616] ima: Allocated hash algorithm: sha1 [ 3.434243] ima: No architecture policies found [ 3.436511] evm: Initialising EVM extended attributes: [ 3.438870] evm: security.selinux [ 3.440052] evm: security.ima [ 3.441393] evm: security.capability [ 3.443094] evm: HMAC attrs: 0x1 [ 3.449168] rtc_cmos 00:05: setting system clock to 2026-09-07 14:20:54 UTC (1788790854) [ 3.456880] debug: unmapping init [mem 0xffffffff98003000-0xffffffff981fffff] [ 3.459461] debug: unmapping init [mem 0xffffffff96d82000-0xffffffff97058fff] [ 3.467553] Write protecting the kernel read-only data: 28672k [ 3.470315] debug: unmapping init [mem 0xffffffff95403000-0xffffffff955fffff] [ 3.473190] debug: unmapping init [mem 0xffffffff95d14000-0xffffffff95dfffff] [ 3.509156] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 3.521156] systemd[1]: Detected virtualization kvm. [ 3.524162] systemd[1]: Detected architecture x86-64. [ 3.526269] systemd[1]: Running in initial RAM disk. Welcome to Rocky Linux 8.10 (Green Obsidian) dracut-049-233.git20240115.el8 (Initramfs)! [ 3.555647] systemd[1]: No hostname configured. [ 3.558121] systemd[1]: Set hostname to . [ 3.560806] random: systemd: uninitialized urandom read (16 bytes read) [ 3.563466] systemd[1]: Initializing machine ID from random generator. [ 3.624716] random: ln: uninitialized urandom read (6 bytes read) [ 3.696872] random: systemd: uninitialized urandom read (16 bytes read) [ 3.699760] systemd[1]: Listening on Journal Socket (/dev/log). [ OK ] Listening on Journal Socket (/dev/log). [ 3.703798] systemd[1]: Reached target Slices. [ OK ] Reached target Slices. [ 3.706850] systemd[1]: Reached target Initrd Root Device. [ OK ] Reached target Initrd Root Device. [ OK ] Reached target Local File Systems. [ OK ] Reached target Swap. [ OK ] Listening on udev Control Socket. [ OK ] Listening on Journal Socket. Starting Create list of required st…ce nodes for the current kernel... Starting Setup Virtual Console... Starting Create Volatile Files and Directories... [ OK ] Started Memstrack Anylazing Service. Starting Apply Kernel Variables... Starting Journal Service... [ OK ] Reached target Timers. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. [ OK ] Listening on udev Kernel Socket. [ OK ] Reached target Sockets. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Started Setup Virtual Console. [ OK ] Started Create Volatile Files and Directories. [ OK ] Started Apply Kernel Variables. Starting dracut cmdline hook... Starting Create Static Device Nodes in /dev... [ OK ] Started Create Static Device Nodes in /dev. [ OK ] Started Journal Service. [ OK ] Started dracut cmdline hook. Starting dracut pre-udev hook... [ 4.297338] device-mapper: uevent: version 1.0.3 [ 4.299632] device-mapper: ioctl: 4.46.0-ioctl (2022-02-22) initialised: dm-devel@redhat.com [ OK ] Started dracut pre-udev hook. Starting udev Kernel Device Manager... [ OK ] Started udev Kernel Device Manager. Starting dracut pre-trigger hook... [ OK ] Started dracut pre-trigger hook. Starting udev Coldplug all Devices... Mounting Kernel Configuration File System... [ OK ] Mounted Kernel Configuration File System. [ OK ] Started udev Coldplug all Devices. [ OK ] Reached target System Initialization. [ OK ] Reached target Basic System. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. Starting dracut initqueue hook... [ 4.961363] random: fast init done [ 4.966571] virtio_net virtio0 ens2: renamed from eth0 [ 5.018331] scsi host0: ata_piix [ 5.038609] scsi host1: ata_piix [ 5.040119] ata1: PATA max MWDMA2 cmd 0x1f0 ctl 0x3f6 bmdma 0xc320 irq 14 [ 5.042360] ata2: PATA max MWDMA2 cmd 0x170 ctl 0x376 bmdma 0xc328 irq 15 [ 9.226024] dracut-initqueue[585]: RTNETLINK answers: File exists [ 9.957384] random: crng init done [ 9.959151] random: 7 urandom warning(s) missed due to ratelimiting Starting nbd nbd0... [ OK ] Started nbd nbd0. [ OK ] Started dracut initqueue hook. Mounting /sysroot... [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. [ 10.181149] EXT4-fs (nbd0): mounted filesystem with ordered data mode. Opts: (null) [ OK ] Mounted /sysroot. [ OK ] Reached target Initrd Root File System. Starting Reload Configuration from the Real Root... [ OK ] Started Reload Configuration from the Real Root. [ OK ] Reached target Initrd File Systems. [ OK ] Reached target Initrd Default Target. Starting dracut pre-pivot and cleanup hook... [ OK ] Started dracut pre-pivot and cleanup hook. Starting Cleaning Up and Shutting Down Daemons... Stopping Hardware RNG Entropy Gatherer Daemon... [ OK ] Stopped target Timers. [ OK ] Stopped dracut pre-pivot and cleanup hook. [ OK ] Stopped target Remote File Systems. [ OK ] Stopped target Remote File Systems (Pre). [ OK ] Stopped target Initrd Default Target. [ OK ] Stopped dracut initqueue hook. [ OK ] Stopped target Initrd Root Device. [ OK ] Stopped Hardware RNG Entropy Gatherer Daemon. [ OK ] Stopped target Basic System. [ OK ] Stopped target Paths. [ OK ] Stopped target Slices. [ OK ] Stopped target Sockets. [ OK ] Stopped target System Initialization. [ OK ] Stopped Create Volatile Files and Directories. [ OK ] Stopped target Local File Systems. [ OK ] Stopped target Swap. [ OK ] Stopped udev Coldplug all Devices. [ OK ] Stopped dracut pre-trigger hook. [ OK ] Stopped target Local Encrypted Volumes. [ OK ] Stopped Dispatch Password Requests to Console Directory Watch. Stopping udev Kernel Device Manager... [ OK ] Stopped Apply Kernel Variables. [ OK ] Started Cleaning Up and Shutting Down Daemons. [ OK ] Stopped udev Kernel Device Manager. [ OK ] Stopped Create Static Device Nodes in /dev. [ OK ] Stopped Create list of required sta…vice nodes for the current kernel. [ OK ] Stopped dracut pre-udev hook. [ OK ] Stopped dracut cmdline hook. [ OK ] Closed udev Control Socket. [ OK ] Closed udev Kernel Socket. Starting Cleanup udevd DB... [ OK ] Started Cleanup udevd DB. [ OK ] Reached target Switch Root. Starting Switch Root... [ 11.237444] printk: systemd: 26 output lines suppressed due to ratelimiting [ 11.467278] SELinux: Disabled at runtime. [ 11.530370] systemd[1]: systemd 239 (239-82.el8_10.5) running in system mode. (+PAM +AUDIT +SELINUX +IMA -APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD +IDN2 -IDN +PCRE2 default-hierarchy=legacy) [ 11.537194] systemd[1]: Detected virtualization kvm. [ 11.539502] systemd[1]: Detected architecture x86-64. Welcome to Rocky Linux 8.10 (Green Obsidian)! [ 11.978523] systemd[1]: initrd-switch-root.service: Succeeded. [ 11.982271] systemd[1]: Stopped Switch Root. [ OK ] Stopped Switch Root. [ 11.986967] systemd[1]: systemd-journald.service: Service has no hold-off time (RestartSec=0), scheduling restart. [ 11.992134] systemd[1]: systemd-journald.service: Scheduled restart job, restart counter is at 1. [ 11.996207] systemd[1]: Stopped Journal Service. [ OK ] Stopped Journal Service. [ 12.002966] systemd[1]: Starting Journal Service... Starting Journal Service... [ 12.007112] systemd[1]: Created slice system-sshd\x2dkeygen.slice. [ OK ] Created slice system-sshd\x2dkeygen.slice. Starting Create list of required st…ce nodes for the current kernel... [ OK ] Reached target rpc_pipefs.target. [ OK ] Started Forward Password Requests to Wall Directory Watch. [ OK ] Created slice User and Session Slice. [ OK ] Created slice system-serial\x2dgetty.slice. Mounting POSIX Message Queue File System... [ OK ] Listening on RPCbind Server Activation Socket. Activating swap /dev/disk/by-label/SWAP... Starting Apply Kernel Variables... [ OK ] Stopped target Switch Root. [ OK ] Stopped target Initrd Root File System. [ OK ] Reached target RPC Port Mapper. Mounting Kernel Debug File System... [ OK ] Listening on udev Kernel Socket. [ OK ] Created slice system-getty.slice. Starting Remount Root and Kernel File Systems... [ 12.089196] Adding 1048572k swap on /dev/vdb. Priority:-2 extents:1 across:1048572k FS [ OK ] Listening on initctl Compatibility Named Pipe. Mounting Huge Pages File System... [ OK ] Stopped target Initrd File Systems. [ OK ] Listening on udev Control Socket. [ OK ] Started Dispatch Password Requests to Console Directory Watch. [ OK ] Reached target Paths. [ OK ] Reached target Local Encrypted Volumes. Starting udev Coldplug all Devices... [ OK ] Reached target Slices. [FAILED] Failed to set up automount Arbitrar…rmats File System Automount Point. See 'systemctl status proc-sys-fs-binfmt_misc.automount' for details. [ OK ] Listening on Process Core Dump Socket. [ OK ] Started Journal Service. [ OK ] Started Create list of required sta…vice nodes for the current kernel. [ OK ] Mounted POSIX Message Queue File System. [ OK ] Activated swap /dev/disk/by-label/SWAP. [ OK ] Started Apply Kernel Variables. [ OK ] Mounted Kernel Debug File System. [FAILED] Failed to start Remount Root and Kernel File Systems. See 'systemctl status systemd-remount-fs.service' for details. [ OK ] Mounted Huge Pages File System. Starting Configure read-only root support... [ OK ] Reached target Swap. Starting Create Static Device Nodes in /dev... Starting Flush Journal to Persistent Storage... [ OK ] Started Flush Journal to Persistent Storage. [ OK ] Started Create Static Device Nodes in /dev. Starting udev Kernel Device Manager... [ OK ] Reached target Local File Systems (Pre). Mounting /home/green/git/lustre-release... Mounting /mnt... [ OK ] Started udev Coldplug all Devices. [ OK ] Mounted /mnt. [ 12.464595] squashfs: version 4.0 (2009/01/31) Phillip Lougher [ OK ] Mounted /home/green/git/lustre-release. [ OK ] Started udev Kernel Device Manager. [ 12.708317] piix4_smbus 0000:00:01.3: SMBus Host Controller at 0x700, revision 0 [ 12.830253] input: PC Speaker as /devices/platform/pcspkr/input/input5 [ 12.908761] RAPL PMU: API unit is 2^-32 Joules, 0 fixed counters, 10737418240 ms ovfl timer [ 12.923505] EDAC sbridge: Ver: 1.1.2 [ 14.939772] Key type dns_resolver registered [ 15.286524] NFS: Registering the id_resolver key type [ 15.291245] Key type id_resolver registered [ 15.293088] Key type id_legacy registered [ OK ] Started Configure read-only root support. Starting Load/Save Random Seed... [ OK ] Reached target Local File Systems. Starting Mark the need to relabel after reboot... Starting Rebuild Dynamic Linker Cache... Starting Create Volatile Files and Directories... [ OK ] Started Load/Save Random Seed. [ OK ] Started Mark the need to relabel after reboot. [ OK ] Started Create Volatile Files and Directories. Starting Update UTMP about System Boot/Shutdown... Starting RPC Bind... [ OK ] Started RPC Bind. [ OK ] Started Update UTMP about System Boot/Shutdown. [ OK ] Started Rebuild Dynamic Linker Cache. Starting Update is Completed... [ OK ] Started Update is Completed. [ OK ] Reached target System Initialization. [ OK ] Started Daily Cleanup of Temporary Directories. [ OK ] Started dnf makecache --timer. [ OK ] Started daily update of the root trust anchor for DNSSEC. [ OK ] Reached target Timers. [ OK ] Listening on D-Bus System Message Bus Socket. [ OK ] Reached target Sockets. [ OK ] Reached target Basic System. Starting Login Service... [ OK ] Started D-Bus System Message Bus. Starting Restore /run/initramfs on shutdown... [ OK ] Started irqbalance daemon. Starting Network Manager... [ OK ] Reached target sshd-keygen.target. [ OK ] Started Hardware RNG Entropy Gatherer Daemon. [ OK ] Started Restore /run/initramfs on shutdown. [ OK ] Started Login Service. [ OK ] Started Network Manager. [ OK ] Reached target Network. Starting OpenSSH server daemon... Starting GSSAPI Proxy Daemon... Starting Dynamic System Tuning Daemon... Starting Network Manager Wait Online... Starting Hostname Service... [ OK ] Started GSSAPI Proxy Daemon. [ OK ] Reached target NFS client services. [ OK ] Reached target Remote File Systems (Pre). [ OK ] Reached target Remote File Systems. Starting Permit User Sessions... [ OK ] Started OpenSSH server daemon. [ OK ] Started Permit User Sessions. [ OK ] Started Serial Getty on ttyS0. [ OK ] Started Getty on tty1. [ OK ] Started Command Scheduler. [ OK ] Started Serial Getty on ttyS1. [ OK ] Reached target Login Prompts. [ OK ] Started Hostname Service. Starting Network Manager Script Dispatcher Service... [ OK ] Started Network Manager Script Dispatcher Service. [ OK ] Started Network Manager Wait Online. [ OK ] Reached target Network is Online. Starting Crash recovery kernel arming... Starting Notify NFS peers of a restart... Starting System Logging Service... [ OK ] Started Notify NFS peers of a restart. [ OK ] Started System Logging Service. Starting Authorization Manager... [ OK ] Started Authorization Manager. [ OK ] Started Dynamic System Tuning Daemon. [ OK ] Reached target Multi-User System. [ OK ] Reached target Graphical Interface. Starting Update UTMP about System Runlevel Changes... [ OK ] Started Update UTMP about System Runlevel Changes. Rocky Linux 8.10 (Green Obsidian) Kernel 4.18.0rh8.10-debug on an x86_64 oleg621-server login: [ 41.915234] libcfs: loading out-of-tree module taints kernel. [ 41.934495] Key type ._llcrypt registered [ 41.935560] Key type .llcrypt registered [ 41.984425] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 53.110932] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 53.874127] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 53.883224] alg: No test for adler32 (adler32-zlib) [ 55.053416] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 55.573837] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 57.215203] Key type lgssc registered [ 58.311301] Lustre: Echo OBD driver; http://www.lustre.org/ [ 59.992140] hrtimer: interrupt took 2201344 ns [ 68.636186] ZFS: Loaded module v2.3.2-1, ZFS pool version 5000, ZFS filesystem version 5 [ 86.130618] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 92.026946] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 92.040830] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 93.137181] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 93.153091] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 93.197751] Lustre: lustre-MDT0000: new disk, initializing [ 93.239746] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 93.250032] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 95.089528] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 101.740325] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 101.781822] Lustre: 6479:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 101.801601] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 101.805898] Lustre: Skipped 1 previous similar message [ 101.853275] Lustre: lustre-MDT0001: new disk, initializing [ 101.881050] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 101.893460] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 101.898952] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 103.529825] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 106.099295] Lustre: Modifying parameter general.debug_raw_pointers=Y in log params [ 109.814928] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 109.919598] Lustre: lustre-OST0000: new disk, initializing [ 109.921593] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 109.924799] Lustre: 8416:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 109.945826] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 112.209699] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 114.042628] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 114.047735] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 114.087169] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 118.890201] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 118.946390] Lustre: lustre-OST0001: new disk, initializing [ 118.949034] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 118.951926] Lustre: 9488:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 118.976992] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 119.671228] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 119.677884] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 119.701233] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 121.425786] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug vfstrace rpctrace dlmtrace neterror ha config ioctl super lfsck all [ 127.787167] Lustre: DEBUG MARKER: Using TIMEOUT=20 [ 133.804947] Lustre: Setting parameter general.lod.*.mdt_hash=crush in log params [ 136.485541] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing check_logdir /tmp/testlogs/ [ 138.398217] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing yml_node [ 140.268717] Lustre: DEBUG MARKER: Client: 2.17.57.82 [ 141.408763] Lustre: DEBUG MARKER: MDS: 2.17.57.82 [ 142.554809] Lustre: DEBUG MARKER: OSS: 2.17.57.82 [ 143.339927] Lustre: DEBUG MARKER: -----============= acceptance-small: conf-sanity ============----- Mon Sep 7 10:23:14 EDT 2026 [ 150.114350] Lustre: DEBUG MARKER: excepting tests: 55 63 67 32newtarball [ 150.829159] Lustre: DEBUG MARKER: skipping tests SLOW=no: 45 69 106 111 114 [ 163.297913] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 163.302824] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 163.307956] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 165.857160] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 165.857457] Lustre: lustre-MDT0000-lwp-OST0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 165.859674] Lustre: Skipped 2 previous similar messages [ 165.867212] Lustre: Skipped 2 previous similar messages [ 169.331631] Lustre: server umount lustre-MDT0000 complete [ 170.890685] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791021 with bad export cookie 8950043253640770242 [ 170.891914] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 170.895984] LustreError: 6470:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 170.975932] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 170.976486] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 170.981782] Lustre: Skipped 1 previous similar message [ 170.984755] Lustre: Skipped 1 previous similar message [ 176.097460] LustreError: 6485:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 176.097855] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 176.108037] LustreError: 6485:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 176.119219] Lustre: Skipped 2 previous similar messages [ 180.191947] LustreError: 6484:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 180.192204] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 180.200519] LustreError: 6484:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 180.203236] Lustre: Skipped 1 previous similar message [ 185.311147] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 185.312466] LustreError: 10062:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 185.322634] LustreError: 10062:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 185.407084] Lustre: server umount lustre-MDT0001 complete [ 187.043933] Lustre: server umount lustre-OST0000 complete [ 188.492746] Lustre: server umount lustre-OST0001 complete [ 191.250254] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 195.593576] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 210.707040] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 210.814815] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 210.827934] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 210.867849] Lustre: lustre-MDT0000: new disk, initializing [ 210.906426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 210.937728] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 212.662705] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 218.334834] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 218.411427] Lustre: 15027:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 218.584189] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 220.560217] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 226.191165] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 229.170883] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 232.607533] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 232.762391] Lustre: lustre-OST0000: new disk, initializing [ 232.765588] Lustre: Skipped 1 previous similar message [ 232.768483] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 232.772296] Lustre: Skipped 2 previous similar messages [ 232.778939] Lustre: 16994:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 232.832305] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 232.835262] Lustre: Skipped 1 previous similar message [ 234.774468] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 234.784100] Lustre: Skipped 1 previous similar message [ 234.787554] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 234.847673] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 235.918791] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 240.611499] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 242.494951] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 242.592756] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 244.358280] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 244.472312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 250.341738] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 250.344588] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 250.349407] LustreError: Skipped 1 previous similar message [ 250.350207] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 250.359865] Lustre: Skipped 1 previous similar message [ 250.369407] Lustre: Skipped 3 previous similar messages [ 251.501859] Lustre: server umount lustre-OST0000 complete [ 254.433982] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 254.440359] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 259.661801] Lustre: server umount lustre-MDT0000 complete [ 260.576728] LustreError: 17752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 260.593446] LustreError: 17752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 261.706074] LustreError: 15020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791112 with bad export cookie 8950043253640771768 [ 261.709941] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 261.711422] LustreError: 15020:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 261.897643] Lustre: server umount lustre-MDT0001 complete [ 264.335739] Lustre: DEBUG MARKER: == conf-sanity test 33a: Mount ost with a large index number ========================================================== 10:25:15 (1788791115) [ 265.583608] Lustre: DEBUG MARKER: SKIP: conf-sanity test_33a mixed loopback and real device not working [ 266.437330] Lustre: DEBUG MARKER: == conf-sanity test 33b: Drop cancel during umount ======= 10:25:17 (1788791117) [ 270.158937] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 270.397833] LustreError: 19401:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 270.446333] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 272.571132] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 277.506425] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 279.916469] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 283.544418] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 286.381575] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 290.269227] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 293.896482] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 297.718059] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 306.145171] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 306.151848] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 306.162654] Lustre: Skipped 1 previous similar message [ 306.166132] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 306.168672] Lustre: Skipped 3 previous similar messages [ 311.292675] Lustre: server umount lustre-OST0000 complete [ 319.358266] Lustre: server umount lustre-MDT0000 complete [ 320.978215] LustreError: 19383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791172 with bad export cookie 8950043253640772657 [ 320.982686] LustreError: 19383:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 320.982731] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 321.115628] Lustre: server umount lustre-MDT0001 complete [ 325.777198] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 327.118138] Key type lgssc unregistered [ 327.274047] LNet: 22932:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 327.277713] LNetError: 22932:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 327.289636] LNet: Removed LNI 192.168.206.121@tcp [ 327.730138] Key type .llcrypt unregistered [ 327.732766] Key type ._llcrypt unregistered [ 338.546892] Key type ._llcrypt registered [ 338.548780] Key type .llcrypt registered [ 338.594537] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 339.003902] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 339.030380] alg: No test for adler32 (adler32-zlib) [ 339.930725] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 340.049439] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 341.648161] Key type lgssc registered [ 342.157558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 345.911503] loop: module loaded [ 345.948713] loop0: detected capacity change from 0 to 204800000 [ 346.037061] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 346.044377] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 346.208033] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 348.920888] loop0: detected capacity change from 0 to 204800000 [ 349.008612] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 349.015506] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 349.022093] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [ 351.386540] loop0: detected capacity change from 0 to 204800000 [ 354.056882] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 354.064536] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 355.176269] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 355.197177] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 355.239447] Lustre: lustre-MDT0000: new disk, initializing [ 355.283389] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 355.293170] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 357.211968] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 361.806993] loop1: detected capacity change from 0 to 204800000 [ 364.344528] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 364.492452] Lustre: lustre-OST07c6: new disk, initializing [ 364.495109] Lustre: srv-lustre-OST07c6: No data found on store. Initialize space. [ 364.499385] Lustre: Skipped 1 previous similar message [ 364.503162] Lustre: 26099:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 364.544830] Lustre: lustre-OST07c6: Imperative Recovery not enabled, recovery window 60-180 [ 367.676642] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 368.548056] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:7c6:ost [ 368.554596] Lustre: cli-lustre-OST07c6-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:7c6:ost] [ 368.584849] Lustre: lustre-OST07c6-osc-MDT0000: update sequence from 0x107c60000 to 0x240000400 [ 371.566445] Lustre: 25084:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 255 < left 264, rollback = 2 [ 371.571592] Lustre: 25084:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/2, destroy: 0/0/0 [ 371.575824] Lustre: 25084:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 3/264/0 [ 371.581405] Lustre: 25084:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 1/11/0, punch: 0/0/0, quota 1/3/0 [ 371.586890] Lustre: 25084:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 4/65/3, delete: 0/0/0 [ 371.590696] Lustre: 25084:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 2/2/0, ref_del: 0/0/0 [ 384.178846] Lustre: 25085:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 250 < left 278, rollback = 2 [ 384.183780] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 384.187788] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 384.192218] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/4 [ 384.195168] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/1, delete: 0/0/0 [ 384.198738] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 388.010369] Lustre: 25085:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 259 < left 278, rollback = 2 [ 388.018253] Lustre: 25085:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 388.022511] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/1, destroy: 0/0/0 [ 388.026920] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 388.031921] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 388.037061] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 388.041252] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 388.045953] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 388.049171] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 388.052916] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 388.055431] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 388.059025] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 391.159199] Lustre: 25085:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 391.165675] Lustre: 25085:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 1 previous similar message [ 391.171444] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 391.177311] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 1 previous similar message [ 391.182508] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 391.186036] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 1 previous similar message [ 391.190042] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 391.195225] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 1 previous similar message [ 391.199112] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 391.202891] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 1 previous similar message [ 391.206906] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 391.210520] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 1 previous similar message [ 398.084604] Lustre: 25086:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 253 < left 278, rollback = 2 [ 398.088522] Lustre: 25086:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 3 previous similar messages [ 398.092089] Lustre: 25086:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/4, destroy: 0/0/0 [ 398.094871] Lustre: 25086:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 398.098533] Lustre: 25086:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 398.101971] Lustre: 25086:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 398.105485] Lustre: 25086:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/1, punch: 0/0/0, quota 4/150/2 [ 398.110182] Lustre: 25086:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 398.113931] Lustre: 25086:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 398.117716] Lustre: 25086:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 398.122141] Lustre: 25086:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 398.125508] Lustre: 25086:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 3 previous similar messages [ 407.397111] Lustre: 25085:0:(osd_internal.h:1460:osd_trans_exec_op()) lustre-MDT0000: opcode 2: before 260 < left 278, rollback = 2 [ 407.401642] Lustre: 25085:0:(osd_internal.h:1460:osd_trans_exec_op()) Skipped 5 previous similar messages [ 407.405060] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) create: 1/4/0, destroy: 0/0/0 [ 407.407900] Lustre: 25085:0:(osd_handler.c:2090:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 407.411662] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) attr_set: 1/1/0, xattr_set: 4/278/0 [ 407.415187] Lustre: 25085:0:(osd_handler.c:2097:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 407.419417] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) write: 2/20/0, punch: 0/0/0, quota 4/150/0 [ 407.423028] Lustre: 25085:0:(osd_handler.c:2107:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 407.426965] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) insert: 2/33/0, delete: 0/0/0 [ 407.430170] Lustre: 25085:0:(osd_handler.c:2114:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 407.432651] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) ref_add: 0/0/0, ref_del: 0/0/0 [ 407.436529] Lustre: 25085:0:(osd_handler.c:2121:osd_trans_dump_creds()) Skipped 5 previous similar messages [ 450.529227] LustreError: lustre-OST07c6-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 450.529380] Lustre: lustre-OST07c6-osc-MDT0000: Connection to lustre-OST07c6 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 450.543618] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 455.648141] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 460.768509] Lustre: lustre-OST07c6: Not available for connect from 0@lo (stopping) [ 463.327545] Lustre: lustre-OST07c6 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 463.404386] Lustre: server umount lustre-OST07c6 complete [ 468.386276] Lustre: server umount lustre-MDT0000 complete [ 476.087647] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 477.430730] Key type lgssc unregistered [ 477.584224] LNet: 28678:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 477.589881] LNetError: 28678:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 477.600464] LNet: Removed LNI 192.168.206.121@tcp [ 477.983114] Key type .llcrypt unregistered [ 477.985705] Key type ._llcrypt unregistered [ 487.537346] Key type ._llcrypt registered [ 487.539120] Key type .llcrypt registered [ 487.589128] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 488.039189] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 488.046774] alg: No test for adler32 (adler32-zlib) [ 488.974936] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 489.089887] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 490.689351] Key type lgssc registered [ 491.241378] Lustre: Echo OBD driver; http://www.lustre.org/ [ 495.058771] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 495.068062] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 496.287516] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 498.406649] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 503.069666] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 503.278102] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 505.542859] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 509.181803] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 511.711630] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 515.010139] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 515.180119] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 517.909079] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 521.324759] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 521.603244] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 536.037651] LustreError: lustre-OST0000-osc-MDT0000: operation ost_setattr to node 0@lo failed: rc = -107 [ 536.038566] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 536.042598] LustreError: Skipped 1 previous similar message [ 536.053539] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 536.248637] Lustre: server umount lustre-OST0000 complete [ 539.106654] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 539.107148] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 539.108942] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 539.119631] Lustre: Skipped 1 previous similar message [ 544.226269] LustreError: 29921:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 544.238062] LustreError: 29921:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 544.249730] Lustre: server umount lustre-MDT0000 complete [ 546.213214] LustreError: 29905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791397 with bad export cookie 6101830963959895250 [ 546.213855] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 546.223824] LustreError: 29905:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 556.583765] Lustre: server umount lustre-MDT0001 complete [ 561.582314] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 562.972643] Key type lgssc unregistered [ 563.106776] LNet: 33662:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 563.111722] LNetError: 33662:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 563.121441] LNet: Removed LNI 192.168.206.121@tcp [ 563.523866] Key type .llcrypt unregistered [ 563.526066] Key type ._llcrypt unregistered [ 572.184297] Key type ._llcrypt registered [ 572.186244] Key type .llcrypt registered [ 572.236846] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 572.605458] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 572.618266] alg: No test for adler32 (adler32-zlib) [ 573.506303] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 573.607296] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 575.215146] Key type lgssc registered [ 575.668831] Lustre: Echo OBD driver; http://www.lustre.org/ [ 579.146765] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 579.156597] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 580.380631] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 582.489663] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 586.998549] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 587.234370] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 589.168573] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 592.537825] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 595.732624] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 599.593262] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 599.820355] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 603.172943] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 604.970503] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 606.827733] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 617.445607] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 617.446136] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 617.456636] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 617.460013] Lustre: Skipped 1 previous similar message [ 618.489591] Lustre: server umount lustre-OST0000 complete [ 622.562162] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 622.563435] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 622.570041] Lustre: Skipped 2 previous similar messages [ 626.562911] Lustre: server umount lustre-MDT0000 complete [ 627.680133] LustreError: 35637:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 627.689508] LustreError: 35637:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 628.307519] LustreError: 36748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791479 with bad export cookie 13403272260014182203 [ 628.312196] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 628.314276] LustreError: 36748:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 628.465631] Lustre: server umount lustre-MDT0001 complete [ 633.329777] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 634.822811] Key type lgssc unregistered [ 634.986719] LNet: 38438:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 634.992568] LNetError: 38438:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 635.003401] LNet: Removed LNI 192.168.206.121@tcp [ 635.473671] Key type .llcrypt unregistered [ 635.475175] Key type ._llcrypt unregistered [ 647.603290] Key type ._llcrypt registered [ 647.604975] Key type .llcrypt registered [ 647.654394] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 648.085829] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 648.141518] alg: No test for adler32 (adler32-zlib) [ 649.044223] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 649.176337] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 650.783155] Key type lgssc registered [ 651.381712] Lustre: Echo OBD driver; http://www.lustre.org/ [ 655.424570] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 655.435606] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 656.678124] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 658.786510] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 663.180798] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 663.369154] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 665.593435] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 669.148381] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 671.488978] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 674.714444] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 674.910390] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 677.539765] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 680.894612] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 681.067395] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 694.240557] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 694.245767] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 694.256933] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 697.720524] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.21@tcp (stopping) [ 698.848944] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 698.850898] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 698.856789] Lustre: Skipped 1 previous similar message [ 698.861972] Lustre: Skipped 2 previous similar messages [ 700.081310] Lustre: server umount lustre-MDT0000 complete [ 701.783434] LustreError: 39811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791552 with bad export cookie 1122146237838746951 [ 701.786664] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 701.787811] LustreError: 39811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 702.828142] LustreError: 39827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 702.834628] LustreError: 39827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 702.837567] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.21@tcp (stopping) [ 703.968574] LustreError: 42324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 703.968898] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 706.976380] LustreError: 42324:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 706.976395] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 706.985384] Lustre: Skipped 1 previous similar message [ 711.072326] LustreError: 41524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 711.080336] LustreError: 41524:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 716.255193] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 716.260182] LustreError: 39827:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 716.260599] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 716.260604] Lustre: Skipped 3 previous similar messages [ 716.285449] LustreError: 39827:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 716.383677] Lustre: server umount lustre-MDT0001 complete [ 728.759586] Lustre: server umount lustre-OST0000 complete [ 735.986494] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 737.496550] Key type lgssc unregistered [ 737.652710] LNet: 43715:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 737.657965] LNetError: 43715:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 737.674709] LNet: Removed LNI 192.168.206.121@tcp [ 738.087642] Key type .llcrypt unregistered [ 738.090174] Key type ._llcrypt unregistered [ 749.631189] Key type ._llcrypt registered [ 749.632535] Key type .llcrypt registered [ 749.676508] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 750.022117] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 750.030070] alg: No test for adler32 (adler32-zlib) [ 750.921109] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 751.040695] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 752.648438] Key type lgssc registered [ 753.129212] Lustre: Echo OBD driver; http://www.lustre.org/ [ 757.252243] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 757.261327] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 758.488375] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 760.349207] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 764.540823] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 764.728512] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 766.641948] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 769.826031] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 772.313763] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 775.411079] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 775.558611] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 778.107689] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 781.255954] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 783.784568] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 795.106248] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 795.107360] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 795.116874] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 795.120224] Lustre: Skipped 1 previous similar message [ 796.251404] Lustre: server umount lustre-OST0000 complete [ 809.504373] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 809.513546] Lustre: Skipped 1 previous similar message [ 809.517513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 809.520945] Lustre: Skipped 1 previous similar message [ 814.564470] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 814.570876] Lustre: Skipped 1 previous similar message [ 815.821188] Lustre: server umount lustre-MDT0000 complete [ 817.781466] LustreError: 45089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791668 with bad export cookie 9927968874257132304 [ 817.785320] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 817.789269] LustreError: 45089:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 824.087355] Lustre: server umount lustre-MDT0001 complete [ 830.227609] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 831.945687] Key type lgssc unregistered [ 832.147473] LNet: 48833:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 832.157241] LNetError: 48833:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 832.166774] LNet: Removed LNI 192.168.206.121@tcp [ 832.835492] Key type .llcrypt unregistered [ 832.837565] Key type ._llcrypt unregistered [ 845.860247] Key type ._llcrypt registered [ 845.862563] Key type .llcrypt registered [ 845.913990] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 846.301367] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 846.355985] alg: No test for adler32 (adler32-zlib) [ 847.259515] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 847.396501] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 849.015135] Key type lgssc registered [ 849.588135] Lustre: Echo OBD driver; http://www.lustre.org/ [ 853.212579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 853.224556] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 854.435133] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 856.344280] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 859.927960] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 860.093378] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 861.772980] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 864.851894] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 867.064766] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 869.604066] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 869.739603] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 872.078585] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 874.982965] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 879.972882] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 882.663378] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 884.210258] Lustre: DEBUG MARKER: Wait for RECONNECT_INTERVAL seconds (10s) [ 895.136657] Lustre: DEBUG MARKER: conf-sanity.sh test_35a 2026-09-0710h35m46s [ 895.939234] Lustre: DEBUG MARKER: Stopping the MDT: lustre-MDT0000 [ 900.578080] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 900.580084] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 900.587192] Lustre: Skipped 2 previous similar messages [ 900.590123] Lustre: Skipped 1 previous similar message [ 901.601451] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 901.604137] Lustre: Skipped 4 previous similar messages [ 907.050714] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.21@tcp (stopping) [ 907.056760] Lustre: Skipped 1 previous similar message [ 911.327114] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 911.444195] Lustre: server umount lustre-MDT0000 complete [ 913.070255] Lustre: DEBUG MARKER: Restarting the MDT: lustre-MDT0000 [ 913.257642] LustreError: 51467:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.272490] LustreError: 51467:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 913.891250] LustreError: 50222:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 913.901160] LustreError: 50222:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 916.085729] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 916.165679] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 916.303331] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 916.345586] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 917.353488] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 918.182961] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 919.728150] Lustre: DEBUG MARKER: Wait for df (36618) ... [ 920.466708] Lustre: DEBUG MARKER: done [ 922.597897] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 922.612245] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 922.615609] Lustre: Skipped 3 previous similar messages [ 927.201209] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 927.208115] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 927.216527] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 928.308368] Lustre: server umount lustre-OST0000 complete [ 932.834102] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 932.834414] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 932.843186] Lustre: Skipped 1 previous similar message [ 936.247083] Lustre: server umount lustre-MDT0000 complete [ 937.987794] LustreError: 50209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791789 with bad export cookie 4703334688187685208 [ 937.991232] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 937.997037] LustreError: 50209:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 938.147693] Lustre: server umount lustre-MDT0001 complete [ 942.853429] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 944.184029] Key type lgssc unregistered [ 944.351482] LNet: 55150:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 944.356225] LNetError: 55150:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 944.368741] LNet: Removed LNI 192.168.206.121@tcp [ 944.794899] Key type .llcrypt unregistered [ 944.796383] Key type ._llcrypt unregistered [ 964.094645] Key type ._llcrypt registered [ 964.095993] Key type .llcrypt registered [ 964.148313] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 964.686539] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 964.699629] alg: No test for adler32 (adler32-zlib) [ 965.584058] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 965.709143] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 967.303156] Key type lgssc registered [ 967.851403] Lustre: Echo OBD driver; http://www.lustre.org/ [ 971.868720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 971.883621] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 973.024182] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 973.038995] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 973.126684] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 975.177781] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 979.346916] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 979.413149] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 979.439428] Lustre: 57336:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 979.573245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 981.459567] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 984.870625] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 987.645265] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 991.022324] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 991.127916] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 991.234670] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 992.557967] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:257) [ 994.465798] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 998.317536] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1002.266408] Lustre: DEBUG MARKER: conf-sanity.sh test_35b 2026-09-0710h37m33s [ 1003.003925] Lustre: DEBUG MARKER: Set up a fake failnode for the MDS [ 1006.323906] Lustre: DEBUG MARKER: Injecting EBUSY on MDS [ 1007.384676] Lustre: DEBUG MARKER: Creating a test file and stat it [ 1007.407735] LustreError: 57342:0:(mdt_handler.c:2379:mdt_getattr_name_lock()) cfs_fail_timeout id 136 sleeping for 40000ms [ 1027.461572] Lustre: lustre-MDT0000: Client 5f5e96c7-f615-43db-bed0-a1d2458e82a5 (at 192.168.206.21@tcp) reconnecting [ 1047.463752] LustreError: 57342:0:(mdt_handler.c:2379:mdt_getattr_name_lock()) cfs_fail_timeout id 136 awake [ 1048.445704] Lustre: DEBUG MARKER: Stop injecting EBUSY on MDS [ 1049.591873] Lustre: DEBUG MARKER: done [ 1053.154231] LustreError: lustre-OST0000-osc-MDT0000: operation ost_destroy to node 0@lo failed: rc = -107 [ 1053.158171] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1053.164113] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1054.180254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1054.181283] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1054.185581] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1057.897763] Lustre: server umount lustre-OST0000 complete [ 1061.344131] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1061.347843] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1061.354444] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1061.357695] Lustre: Skipped 1 previous similar message [ 1065.443072] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1065.447832] Lustre: Skipped 4 previous similar messages [ 1065.780243] Lustre: server umount lustre-MDT0000 complete [ 1067.611963] LustreError: 57326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788791918 with bad export cookie 6507219431430937786 [ 1067.615790] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1067.617574] LustreError: 57326:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1067.794524] Lustre: server umount lustre-MDT0001 complete [ 1072.802918] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1074.013978] Key type lgssc unregistered [ 1074.171590] LNet: 61661:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1074.177527] LNetError: 61661:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1074.186990] LNet: Removed LNI 192.168.206.121@tcp [ 1074.549239] Key type .llcrypt unregistered [ 1074.551310] Key type ._llcrypt unregistered [ 1094.749631] Key type ._llcrypt registered [ 1094.751649] Key type .llcrypt registered [ 1094.800487] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1095.210650] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1095.248404] alg: No test for adler32 (adler32-zlib) [ 1096.138715] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 1096.255226] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1097.864778] Key type lgssc registered [ 1098.359232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1099.856636] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1099.866659] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1101.009737] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 1101.021960] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 1101.111809] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1101.567217] Lustre: Failing over lustre-MDT0000 [ 1101.696644] Lustre: server umount lustre-MDT0000 complete [ 1104.498736] Lustre: DEBUG MARKER: == conf-sanity test 38: MDS recreates missing lov_objid file from OST data ========================================================== 10:39:15 (1788791955) [ 1107.268139] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1107.461166] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1109.270763] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1113.137799] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1113.197355] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 1113.211161] Lustre: 64433:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1113.354687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1115.188354] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1118.589971] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1121.219503] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1125.166679] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1125.273600] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 1125.399401] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1128.448413] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1132.074932] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1133.605692] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:259 to 0x280000401:289) [ 1141.774259] Lustre: DEBUG MARKER: copying 10 files to /mnt/lustre/d38.conf-sanity [ 1148.898114] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1148.898781] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1148.903764] Lustre: Skipped 1 previous similar message [ 1150.505910] Lustre: server umount lustre-MDT0000 complete [ 1152.026183] LustreError: 65169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788792003 with bad export cookie 11259216413007686251 [ 1152.029949] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1152.030884] LustreError: 65169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1152.153297] Lustre: server umount lustre-MDT0001 complete [ 1153.514249] Lustre: DEBUG MARKER: delete lov_objid file on MDS [ 1155.433223] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1158.874267] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1162.144261] LustreError: 63357:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@000000003f7f4e71 x1875684304652160/t0(0) o250->MGC192.168.206.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1162.383384] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1162.442258] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000402 [ 1164.334774] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1167.842860] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1167.843165] LustreError: 66283:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1167.851346] Lustre: Skipped 1 previous similar message [ 1167.859883] LustreError: 66283:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1168.311032] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1169.509818] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1170.255301] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1170.527683] Lustre: 63361:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788792005/real 1788792005] req@000000008339d966 x1875684304651776/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788792021 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1173.600676] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1173.967202] Lustre: 63358:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788792008/real 1788792008] req@00000000137cab27 x1875684304651904/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788792024 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1176.031340] Lustre: 63359:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788792010/real 1788792010] req@00000000391620f0 x1875684304652032/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788792026 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1176.058684] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1178.079115] Lustre: 63360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788792013/real 1788792013] req@000000003368368a x1875684304652288/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788792029 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1178.592510] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1178.596046] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1178.601809] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1178.605390] Lustre: Skipped 2 previous similar messages [ 1180.133188] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1180.135859] Lustre: Skipped 2 previous similar messages [ 1184.477659] Lustre: server umount lustre-MDT0000 complete [ 1185.252520] LustreError: 68199:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1185.268960] LustreError: 68199:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1186.037048] LustreError: 65169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788792037 with bad export cookie 11259216413007688519 [ 1186.040019] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1186.041325] LustreError: 65169:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1186.167562] Lustre: server umount lustre-MDT0001 complete [ 1189.259210] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1193.714313] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1196.512604] LustreError: 63357:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000f4ff6270 x1875684304673664/t0(0) o250->MGC192.168.206.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1196.680286] LustreError: 66284:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1196.722539] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1196.727032] Lustre: Skipped 1 previous similar message [ 1196.763574] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:3 to 0x280000402:33) [ 1198.810057] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1202.145992] LustreError: 68929:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1202.146693] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1202.163474] Lustre: Skipped 2 previous similar messages [ 1203.230683] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1205.362139] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1206.495175] Lustre: 63360:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788792041/real 1788792041] req@000000003368368a x1875684304673280/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788792057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 1208.664409] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1208.808248] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1208.811981] Lustre: Skipped 1 previous similar message [ 1210.858523] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 1211.201184] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1213.921156] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1213.926141] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1213.935892] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1221.088296] LustreError: 71604:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1221.097915] LustreError: 71604:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1222.003481] Lustre: server umount lustre-MDT0000 complete [ 1223.620667] LustreError: 65170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788792074 with bad export cookie 11259216413007689898 [ 1223.625427] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1223.630435] LustreError: 65170:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 1223.779885] Lustre: server umount lustre-MDT0001 complete [ 1226.542896] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 1227.984312] Lustre: DEBUG MARKER: files compared the same [ 1238.134899] Lustre: server umount lustre-OST0000 complete [ 1245.619841] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1246.855487] Key type lgssc unregistered [ 1246.999539] LNet: 74147:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1247.003233] LNetError: 74147:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1247.015536] LNet: Removed LNI 192.168.206.121@tcp [ 1247.388369] Key type .llcrypt unregistered [ 1247.390102] Key type ._llcrypt unregistered [ 1257.889561] Key type ._llcrypt registered [ 1257.891339] Key type .llcrypt registered [ 1257.935768] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1258.332575] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1258.386557] alg: No test for adler32 (adler32-zlib) [ 1259.317694] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 1259.437761] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1261.031142] Key type lgssc registered [ 1261.536579] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1264.810121] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1264.818221] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1266.024565] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1267.749941] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug +malloc all [ 1271.521953] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1271.683909] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1273.453510] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug +malloc all [ 1276.735952] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1279.285813] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1282.525818] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1282.687108] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1285.484166] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug +malloc all [ 1288.717664] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1291.941523] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:65) [ 1307.616609] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1307.616663] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1307.622417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1307.629926] Lustre: Skipped 1 previous similar message [ 1312.736588] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1312.741391] Lustre: Skipped 1 previous similar message [ 1317.343111] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 1317.400853] Lustre: server umount lustre-OST0000 complete [ 1322.977382] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1322.977646] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1322.981644] LustreError: Skipped 1 previous similar message [ 1322.987430] Lustre: Skipped 1 previous similar message [ 1322.987943] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1322.994591] Lustre: Skipped 1 previous similar message [ 1325.168398] Lustre: server umount lustre-MDT0000 complete [ 1326.741989] LustreError: 75521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788792177 with bad export cookie 10142767122205559049 [ 1326.743690] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1326.746016] LustreError: 75521:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 1326.851194] Lustre: server umount lustre-MDT0001 complete [ 1331.149519] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1332.456672] Key type lgssc unregistered [ 1332.607574] LNet: 79094:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1332.611906] LNetError: 79094:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1332.621495] LNet: Removed LNI 192.168.206.121@tcp [ 1333.036245] Key type .llcrypt unregistered [ 1333.038156] Key type ._llcrypt unregistered [ 1344.431612] Key type ._llcrypt registered [ 1344.433352] Key type .llcrypt registered [ 1344.493619] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1345.012203] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1345.047132] alg: No test for adler32 (adler32-zlib) [ 1346.024663] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 1346.148680] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1347.791882] Key type lgssc registered [ 1348.491214] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1354.092172] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1354.125875] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1369.695464] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [ 1374.815377] LustreError: 80461:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.121@tcp: failed processing log, type 4: rc = -110 [ 1401.503663] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1408.967762] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1419.006456] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1429.154704] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1429.356886] Lustre: *** cfs_fail_loc=706, val=0*** [ 1429.964727] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1430.084944] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:97) [ 1435.107479] LustreError: 80486:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1435.169908] LustreError: 80486:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1436.137035] LustreError: 81546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1436.145445] LustreError: 81546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1439.441225] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1441.254077] LustreError: 81546:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1441.318340] LustreError: 81546:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1446.393815] LustreError: 81547:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1446.438883] LustreError: 81547:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1451.496706] LustreError: 81752:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1451.535677] LustreError: 81752:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1455.692653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1456.774481] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1462.171741] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1470.847421] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1475.948152] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1481.185421] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1481.198072] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1481.224217] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1482.215214] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 1482.226676] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1482.240828] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1484.284654] Lustre: server umount lustre-OST0000 complete [ 1492.448213] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1492.458563] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1492.477929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1492.482791] Lustre: Skipped 1 previous similar message [ 1494.655629] Lustre: server umount lustre-MDT0000 complete [ 1498.082123] LustreError: 81550:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1498.095208] LustreError: 81550:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1498.755553] LustreError: 80468:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788792349 with bad export cookie 3536482988241371011 [ 1498.770195] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1499.189968] Lustre: server umount lustre-MDT0001 complete [ 1510.476524] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1512.963267] Key type lgssc unregistered [ 1513.238673] LNet: 84084:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1513.256666] LNetError: 84084:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1514.276796] LNet: Removed LNI 192.168.206.121@tcp [ 1515.119955] Key type .llcrypt unregistered [ 1515.123606] Key type ._llcrypt unregistered [ 1536.165940] Key type ._llcrypt registered [ 1536.168327] Key type .llcrypt registered [ 1536.269937] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1537.424026] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1537.455286] alg: No test for adler32 (adler32-zlib) [ 1538.510966] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 1538.719545] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1540.351156] Key type lgssc registered [ 1541.465645] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1549.343917] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1549.371345] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1554.878027] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1562.707847] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1563.236349] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1567.796283] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1568.225581] LustreError: 86187:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1568.247560] LustreError: 86187:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1573.347399] LustreError: 86185:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1573.362457] LustreError: 86185:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1575.846318] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1576.221206] LustreError: 86186:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1576.241230] LustreError: 86186:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1576.295801] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1581.418793] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1581.538765] LustreError: 86190:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1581.559045] LustreError: 86190:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 1586.666323] LustreError: 86191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1586.681655] LustreError: 86191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1589.025772] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1589.037567] Lustre: MGS: Client cf5fbb32-78ea-47b4-88f8-06121ba40d20 (at 0@lo) reconnecting [ 1589.061747] LustreError: 86201:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1589.260231] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1589.319565] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000402:35 to 0x280000402:129) [ 1593.300644] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1609.707304] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1609.709396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1609.718597] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1609.724111] Lustre: Skipped 1 previous similar message [ 1613.152435] Lustre: server umount lustre-OST0000 complete [ 1619.939682] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1619.941250] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1619.969681] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1619.984553] Lustre: Skipped 2 previous similar messages [ 1622.824270] Lustre: server umount lustre-MDT0000 complete [ 1625.064601] LustreError: 86191:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1625.090260] LustreError: 86191:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 1626.995791] Lustre: server umount lustre-MDT0001 complete [ 1630.742222] Lustre: server umount MGS complete [ 1641.548827] Lustre: DEBUG MARKER: == conf-sanity test 41b: mount mds with --nosvc and --nomgs on first mount ========================================================== 10:48:12 (1788792492) [ 1672.380955] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 1687.152504] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1725.910211] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1730.727566] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1739.705188] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1739.921901] Lustre: Setting parameter lustre-MDT0001.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0001 [ 1739.961288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 1740.030505] Lustre: lustre-MDT0001: new disk, initializing [ 1740.114410] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1740.131949] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1740.136115] Lustre: 93392:0:(lod_sub_object.c:951:lod_sub_prep_llog()) lustre-MDT0001-mdtlov: can't create new llog: rc = -115 [ 1740.140825] LustreError: 93392:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osd: get update log duration 0, retries 0, failed: rc = -115 [ 1744.825597] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1755.209724] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1755.465608] Lustre: lustre-OST0000: new disk, initializing [ 1755.476602] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 1755.483528] Lustre: 94158:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 1755.566982] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1761.610983] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1762.922829] Lustre: 94526:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@00000000de72a392 x1875684768128384/t0(0) o700->lustre-OST0000-osc-MDT0001@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788792629 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1771.165460] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 1777.786670] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 1777.800530] Lustre: MGS: Client 5e24bf13-64de-4aab-ba8b-5c641f24e50f (at 0@lo) reconnecting [ 1777.816213] LustreError: 93378:0:(mgs_handler.c:244:mgs_revoke_lock()) MGS: can't take cfg lock for 0x65727473756c/0x2 : rc = -11 [ 1777.854742] Lustre: 92662:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0000/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 1777.891743] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 1777.978983] Lustre: lustre-MDT0000: new disk, initializing [ 1778.181426] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1778.217890] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 1778.235913] Lustre: srv-lustre-MDT0001: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1779.298987] Lustre: 95227:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000004501f616 x1875684768135424/t0(0) o700->lustre-MDT0001-osp-MDT0000@0@lo:30/10 lens 264/248 e 0 to 0 dl 1788792646 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [ 1782.521756] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1784.479706] Lustre: srv-lustre-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [ 1788.401565] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 1788.432913] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 1788.570883] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000400 [ 1814.496271] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 1814.518273] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1814.549808] Lustre: Skipped 1 previous similar message [ 1814.566427] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.082920] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 1817.118505] Lustre: Skipped 2 previous similar messages [ 1821.183735] Lustre: server umount lustre-OST0000 complete [ 1827.302856] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 1827.325866] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 1827.328564] Lustre: Skipped 2 previous similar messages [ 1827.330872] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 1827.330879] LustreError: Skipped 1 previous similar message [ 1827.378360] Lustre: Skipped 1 previous similar message [ 1832.749492] Lustre: server umount lustre-MDT0000 complete [ 1837.543744] LustreError: 93368:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1837.573176] LustreError: 93368:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 1846.105725] Lustre: server umount lustre-MDT0001 complete [ 1853.239254] Lustre: server umount MGS complete [ 1873.248564] Lustre: DEBUG MARKER: == conf-sanity test 41c: concurrent mounts of MDT/OST should all fail but one ========================================================== 10:52:02 (1788792722) [ 1898.991694] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 1904.187330] Key type lgssc unregistered [ 1904.924263] LNet: 97733:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 1904.936169] LNetError: 97733:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 1904.961977] LNet: Removed LNI 192.168.206.121@tcp [ 1907.100328] Key type .llcrypt unregistered [ 1907.102955] Key type ._llcrypt unregistered [ 1929.183801] Key type ._llcrypt registered [ 1929.185933] Key type .llcrypt registered [ 1929.331635] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 1930.663871] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 1930.720399] alg: No test for adler32 (adler32-zlib) [ 1931.905867] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 1932.135896] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 1933.815247] Key type lgssc registered [ 1935.016657] Lustre: Echo OBD driver; http://www.lustre.org/ [ 1943.842259] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 1943.857626] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1945.354402] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 1949.617856] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1955.810531] LustreError: 98957:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1955.834711] LustreError: 98957:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 1958.037742] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 1958.443509] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 1963.207975] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1970.664777] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 1977.567947] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 1983.718105] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 1983.911439] LustreError: 100778:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 1984.040700] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 1990.125465] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:33) [ 1990.466768] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 1997.895937] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2009.568268] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2009.582670] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2009.600119] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2010.597269] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2010.603554] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2011.816614] Lustre: server umount lustre-OST0000 complete [ 2015.714216] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2015.716888] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2015.724663] Lustre: Skipped 1 previous similar message [ 2015.734889] Lustre: Skipped 2 previous similar messages [ 2020.843824] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2020.851539] Lustre: Skipped 1 previous similar message [ 2021.504793] Lustre: server umount lustre-MDT0000 complete [ 2025.354567] LustreError: 98937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788792876 with bad export cookie 3046363574553631984 [ 2025.368574] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2025.372047] LustreError: 98937:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2025.624317] Lustre: server umount lustre-MDT0001 complete [ 2035.571768] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 2038.052264] Key type lgssc unregistered [ 2038.346768] LNet: 102491:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2038.354798] LNetError: 102491:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2038.372566] LNet: Removed LNI 192.168.206.121@tcp [ 2039.102605] Key type .llcrypt unregistered [ 2039.108225] Key type ._llcrypt unregistered [ 2054.302333] Key type ._llcrypt registered [ 2054.305212] Key type .llcrypt registered [ 2054.435899] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2055.323886] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2055.376184] alg: No test for adler32 (adler32-zlib) [ 2056.652978] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 2056.895287] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 2058.615435] Key type lgssc registered [ 2059.945558] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2066.037621] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2066.043629] LustreError: 103574:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2066.053547] LustreError: 103575:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2066.058899] LustreError: 103574:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4993 [ 2066.065831] LustreError: 103574:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -114 [ 2066.071760] LustreError: 103574:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2066.076049] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2067.554268] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2076.033659] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2076.213222] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2081.817217] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2089.291823] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing lsmod [ 2092.941491] LustreError: 104782:0:(tgt_mount.c:2570:server_fill_super()) cfs_race id 716 sleeping [ 2092.945187] LustreError: 104780:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 waking [ 2092.959194] LustreError: 104782:0:(tgt_mount.c:2570:server_fill_super()) cfs_fail_race id 716 awake: rc=4986 [ 2092.976908] LustreError: 104782:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -114 [ 2092.983809] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2092.988854] LustreError: 104782:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -114 [ 2093.317482] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2098.658922] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2098.662746] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2098.675221] Lustre: Skipped 2 previous similar messages [ 2098.681016] Lustre: Skipped 1 previous similar message [ 2101.796755] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2101.831729] Lustre: Skipped 3 previous similar messages [ 2103.371505] Lustre: server umount lustre-MDT0000 complete [ 2106.850209] LustreError: 103612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2106.879733] LustreError: 103612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 2107.252327] LustreError: 103591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788792958 with bad export cookie 9799237755161991198 [ 2107.268527] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2107.273262] LustreError: 103591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2108.063288] LustreError: 104044:0:(client.c:1394:ptlrpc_import_delay_req()) @@@ IMP_CLOSED req@00000000af06a5ea x1875685311272448/t0(0) o900->lustre-MDT0000-lwp-MDT0001@0@lo:29/10 lens 264/248 e 0 to 0 dl 0 ref 2 fl Rpc:QU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 2108.360467] Lustre: server umount lustre-MDT0001 complete [ 2124.222926] Lustre: server umount lustre-OST0000 complete [ 2134.710364] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2135.359806] LustreError: 105849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2135.470100] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2140.577683] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2140.654369] LustreError: 105850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2145.761167] LustreError: 105849:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2151.137690] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2151.624612] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2156.479196] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2165.841532] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2173.490420] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2182.657542] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2183.212725] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2189.848352] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2197.603908] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2206.759390] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:3 to 0x280000400:65) [ 2216.939279] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2216.957250] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2216.981330] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2216.991798] LustreError: Skipped 1 previous similar message [ 2217.017577] Lustre: Skipped 1 previous similar message [ 2221.335205] Lustre: server umount lustre-OST0000 complete [ 2227.169812] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2227.192246] Lustre: Skipped 1 previous similar message [ 2227.201362] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2232.293418] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2232.308613] Lustre: Skipped 2 previous similar messages [ 2233.079149] Lustre: server umount lustre-MDT0000 complete [ 2237.409096] LustreError: 105844:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2237.434387] LustreError: 105844:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2239.539221] LustreError: 105830:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788793090 with bad export cookie 9799237755161991898 [ 2239.563333] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2240.259394] Lustre: server umount lustre-MDT0001 complete [ 2254.523737] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 2257.444119] Key type lgssc unregistered [ 2257.903867] LNet: 109410:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2257.911656] LNetError: 109410:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2258.981813] LNet: Removed LNI 192.168.206.121@tcp [ 2260.226927] Key type .llcrypt unregistered [ 2260.228939] Key type ._llcrypt unregistered [ 2294.550044] Key type ._llcrypt registered [ 2294.559626] Key type .llcrypt registered [ 2294.764706] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2297.270956] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2297.500493] alg: No test for adler32 (adler32-zlib) [ 2298.777649] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 2299.362560] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 2301.039149] Key type lgssc registered [ 2302.325727] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2311.995244] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2312.040721] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2313.582798] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2319.862276] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2323.937028] LustreError: 110838:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2323.956584] LustreError: 110838:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2329.059279] LustreError: 110839:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2332.141273] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2332.798406] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2339.163373] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2346.949798] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2352.599741] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2361.815933] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2362.236973] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2369.078560] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2377.697734] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2384.819954] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:67 to 0x280000400:97) [ 2393.126339] Lustre: Setting parameter lustre-client.llite.some_wrong_param=10 in log lustre-client [ 2397.556286] Lustre: Modifying parameter lustre-client.llite.some_wrong_param=20 in log lustre-client [ 2405.356728] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2405.359669] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2405.363178] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2405.376112] Lustre: Skipped 1 previous similar message [ 2410.483936] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2410.499321] Lustre: Skipped 1 previous similar message [ 2410.501321] LustreError: 112659:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2411.075889] Lustre: server umount lustre-OST0000 complete [ 2419.680965] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2419.699153] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2419.735276] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2420.707561] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2422.274582] Lustre: server umount lustre-MDT0000 complete [ 2425.831620] LustreError: 110833:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2425.868216] LustreError: 110833:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2426.890672] LustreError: 110818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788793277 with bad export cookie 229039766297943517 [ 2426.893791] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2426.906294] LustreError: 110818:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 2427.387657] Lustre: server umount lustre-MDT0001 complete [ 2440.830948] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 2444.525813] Key type lgssc unregistered [ 2444.844066] LNet: 114516:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2444.854984] LNetError: 114516:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2444.874509] LNet: Removed LNI 192.168.206.121@tcp [ 2445.921335] Key type .llcrypt unregistered [ 2445.923833] Key type ._llcrypt unregistered [ 2464.649388] Key type ._llcrypt registered [ 2464.653830] Key type .llcrypt registered [ 2464.779770] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2465.723352] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2465.802043] alg: No test for adler32 (adler32-zlib) [ 2466.837970] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 2467.018471] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 2468.671381] Key type lgssc registered [ 2470.044248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2478.180449] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2478.194349] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2479.626555] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2484.758448] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2489.827442] LustreError: 115743:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2489.842685] LustreError: 115743:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2493.620105] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2493.927648] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2498.977319] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2507.369069] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2514.008674] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2522.744706] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2523.265062] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2524.291120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:99 to 0x280000400:129) [ 2531.766220] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2538.992304] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2549.217647] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 2549.220533] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2549.229424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2549.249943] Lustre: Skipped 1 previous similar message [ 2549.268286] Lustre: Skipped 1 previous similar message [ 2552.976351] Lustre: server umount lustre-OST0000 complete [ 2559.456920] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2559.460158] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2559.485316] Lustre: Skipped 1 previous similar message [ 2559.495689] Lustre: Skipped 1 previous similar message [ 2563.846896] Lustre: server umount lustre-MDT0000 complete [ 2564.580293] LustreError: 115742:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2564.610707] LustreError: 115742:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2568.869657] LustreError: 115723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788793419 with bad export cookie 6534835508965102515 [ 2568.881052] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2568.894201] LustreError: 115723:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2569.389255] Lustre: server umount lustre-MDT0001 complete [ 2581.050860] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 2584.229924] Key type lgssc unregistered [ 2584.477829] LNet: 119280:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2584.485330] LNetError: 119280:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2584.506167] LNet: Removed LNI 192.168.206.121@tcp [ 2585.381931] Key type .llcrypt unregistered [ 2585.383651] Key type ._llcrypt unregistered [ 2609.046383] Key type ._llcrypt registered [ 2609.049228] Key type .llcrypt registered [ 2609.191666] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2610.363408] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2610.401817] alg: No test for adler32 (adler32-zlib) [ 2611.608446] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 2611.986326] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 2613.743171] Key type lgssc registered [ 2615.044704] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2623.451510] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2623.476798] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2624.984535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2629.954599] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2635.234463] LustreError: 120701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2635.261771] LustreError: 120701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2638.849840] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2639.247321] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2643.592089] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2651.236906] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2656.827514] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2662.863366] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2663.291896] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2669.652653] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2670.459427] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 2677.193030] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2684.693923] Lustre: Setting parameter lustre-MDT0000.mdt.root_squash=0:0 in log lustre-MDT0000 [ 2688.708893] Lustre: Setting parameter lustre-MDT0000.mdt.nosquash_nids=NONE in log lustre-MDT0000 [ 2688.721810] Lustre: Skipped 2 previous similar messages [ 2695.355060] Lustre: lustre-MDT0001: root_squash is set to 0:0 [ 2695.446641] Lustre: lustre-MDT0001: nosquash_nids is cleared [ 2695.492934] Lustre: Modifying parameter lustre-MDT0000.mdt.root_squash=500:500 in log lustre-MDT0000 [ 2695.515422] Lustre: Skipped 2 previous similar messages [ 2705.455108] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2705.466092] Lustre: Skipped 1 previous similar message [ 2711.984329] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=2@gni 0@lo 192.168.206.21@tcp 192.168.0.[2,10]@tcp in log lustre-MDT0000 [ 2712.004679] Lustre: Skipped 2 previous similar messages [ 2712.959503] Lustre: lustre-MDT0001: nosquash_nids set to 2@gni 0@lo 192.168.206.21@tcp 192.168.0.[2,10]@tcp [ 2725.501901] Lustre: Modifying parameter lustre-MDT0000.mdt.nosquash_nids=192.168.206.21/24@tcp in log lustre-MDT0000 [ 2725.539232] Lustre: Skipped 2 previous similar messages [ 2734.644163] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.21/24@tcp [ 2734.650696] Lustre: Skipped 1 previous similar message [ 2742.256371] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2742.267907] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2742.274842] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2742.311753] Lustre: Skipped 1 previous similar message [ 2747.360451] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2747.376441] Lustre: Skipped 2 previous similar messages [ 2748.199565] Lustre: server umount lustre-OST0000 complete [ 2757.138957] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 2757.154735] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2757.221629] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2761.480247] Lustre: server umount lustre-MDT0000 complete [ 2762.721579] LustreError: 120701:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2762.776565] LustreError: 120701:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2767.840974] LustreError: 120696:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2767.895952] LustreError: 120696:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2768.103395] LustreError: 123330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788793619 with bad export cookie 6709462650339884426 [ 2768.110714] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2768.116582] LustreError: 123330:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 2769.153144] Lustre: server umount lustre-MDT0001 complete [ 2785.358507] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 2788.561616] Key type lgssc unregistered [ 2788.873852] LNet: 125649:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2788.884443] LNetError: 125649:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2789.944771] LNet: Removed LNI 192.168.206.121@tcp [ 2791.008359] Key type .llcrypt unregistered [ 2791.009674] Key type ._llcrypt unregistered [ 2821.514883] Key type ._llcrypt registered [ 2821.518405] Key type .llcrypt registered [ 2821.629993] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2822.633246] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2822.726590] alg: No test for adler32 (adler32-zlib) [ 2823.938736] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 2824.208769] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 2825.935164] Key type lgssc registered [ 2827.174284] Lustre: Echo OBD driver; http://www.lustre.org/ [ 2835.829972] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 2835.867585] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2837.480882] Lustre: lustre-MDT0000: root_squash is set to 500:500 [ 2837.511438] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 2837.528615] Lustre: lustre-MDT0000: nosquash_nids set to 192.168.206.21/24@tcp [ 2843.852829] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2847.718406] LustreError: 127220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2847.744185] LustreError: 127220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 2852.852642] LustreError: 127221:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2854.066319] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 2854.401870] Lustre: lustre-MDT0001: root_squash is set to 500:500 [ 2854.448578] Lustre: lustre-MDT0001: nosquash_nids set to 192.168.206.21/24@tcp [ 2854.524357] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 2859.260212] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2866.784263] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 2872.831595] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 2880.067392] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 2880.524970] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 2882.631918] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000400:169 to 0x280000400:193) [ 2887.523645] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 2896.537767] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 2908.128511] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 2908.135564] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2908.154098] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2909.155598] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2909.159686] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 2909.181255] Lustre: Skipped 1 previous similar message [ 2914.190705] Lustre: server umount lustre-OST0000 complete [ 2919.407584] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 2919.409117] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2919.424333] Lustre: Skipped 1 previous similar message [ 2919.444184] Lustre: Skipped 1 previous similar message [ 2924.517790] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2924.523769] Lustre: Skipped 1 previous similar message [ 2929.634034] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 2929.644426] Lustre: Skipped 1 previous similar message [ 2932.191215] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 2932.387346] Lustre: server umount lustre-MDT0000 complete [ 2934.754818] LustreError: 127220:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 2934.778238] LustreError: 127220:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 2936.529786] LustreError: 127201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788793787 with bad export cookie 2074317299074722523 [ 2936.535878] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 2936.540662] LustreError: 127201:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 2936.915907] Lustre: server umount lustre-MDT0001 complete [ 2948.215435] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 2951.318325] Key type lgssc unregistered [ 2951.636804] LNet: 130810:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 2951.649194] LNetError: 130810:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 2951.669909] LNet: Removed LNI 192.168.206.121@tcp [ 2952.500322] Key type .llcrypt unregistered [ 2952.502670] Key type ._llcrypt unregistered [ 2979.277926] Key type ._llcrypt registered [ 2979.280107] Key type .llcrypt registered [ 2979.404322] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 2994.500862] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 2995.686891] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 2995.700064] alg: No test for adler32 (adler32-zlib) [ 2996.978561] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 2997.277235] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 2999.023119] Key type lgssc registered [ 3000.259234] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3038.314853] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3038.343679] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3039.613839] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3039.660593] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3039.815773] Lustre: lustre-MDT0000: new disk, initializing [ 3039.931049] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3039.947145] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3044.433595] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3054.175134] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3054.275290] Lustre: 134228:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3054.333828] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3054.337980] Lustre: Skipped 1 previous similar message [ 3054.429536] Lustre: lustre-MDT0001: new disk, initializing [ 3054.532579] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3054.575301] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3054.585674] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3059.103246] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3069.805933] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3075.899184] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3083.129821] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3083.423240] Lustre: lustre-OST0000: new disk, initializing [ 3083.430875] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3083.449840] Lustre: 136194:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3083.542986] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3089.449948] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3089.461397] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3089.577187] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3089.721691] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3100.096296] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3105.561590] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3105.798312] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3111.490119] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3111.791451] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3114.977909] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 3114.998906] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3115.017662] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3115.495687] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3120.549516] Lustre: server umount lustre-OST0000 complete [ 3125.736814] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3125.762113] Lustre: Skipped 1 previous similar message [ 3125.769030] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3125.782617] Lustre: Skipped 3 previous similar messages [ 3130.850489] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3130.862048] Lustre: Skipped 1 previous similar message [ 3131.647975] Lustre: server umount lustre-MDT0000 complete [ 3136.270694] LustreError: 134218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788793987 with bad export cookie 3113554659619095274 [ 3136.274595] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3136.286134] LustreError: 134218:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3136.872994] Lustre: server umount lustre-MDT0001 complete [ 3146.432870] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3146.943294] LustreError: 138413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3146.969493] LustreError: 138413:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3147.101802] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3151.823583] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3152.353208] LustreError: 138414:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3152.385166] LustreError: 138414:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3157.477691] LustreError: 138413:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3160.260192] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3160.763042] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3165.316575] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3172.988695] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3178.536928] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3186.981259] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3187.500937] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3194.584661] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3203.473880] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3209.374916] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 3209.612993] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3214.709839] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 3215.019975] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3225.465685] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3225.676653] Lustre: lustre-OST0001: new disk, initializing [ 3225.686706] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 3225.701832] Lustre: 141785:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3225.809474] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 3231.867966] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 3231.885510] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 3231.996266] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 3232.216805] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3241.520466] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 3241.732447] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 3247.160624] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 3247.458885] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 3254.988362] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8af951ffb800.ost_server_uuid 50 [ 3256.919510] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8af951ffb800.ost_server_uuid in IDLE state after 0 sec [ 3268.065214] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3268.067213] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3268.089639] LustreError: Skipped 1 previous similar message [ 3268.099812] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3268.133441] Lustre: Skipped 2 previous similar messages [ 3268.190404] Lustre: Skipped 1 previous similar message [ 3273.204688] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 3273.220843] Lustre: Skipped 1 previous similar message [ 3273.809427] Lustre: server umount lustre-OST0001 complete [ 3278.307871] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3278.309622] LustreError: 140237:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3278.327393] Lustre: Skipped 1 previous similar message [ 3278.366765] LustreError: 140237:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3283.451080] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3283.451846] LustreError: 140239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3283.462186] Lustre: Skipped 3 previous similar messages [ 3283.491852] LustreError: 140239:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3284.348211] Lustre: server umount lustre-OST0000 complete [ 3293.667343] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3293.673312] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3293.691931] Lustre: Skipped 1 previous similar message [ 3303.391265] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 3303.559075] Lustre: server umount lustre-MDT0000 complete [ 3303.906852] LustreError: 139612:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3303.929199] LustreError: 139612:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 3307.346200] LustreError: 138393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788794158 with bad export cookie 3113554659619096156 [ 3307.348660] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3307.354853] LustreError: 138393:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 3307.761386] Lustre: server umount lustre-MDT0001 complete [ 3326.971869] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 3330.361808] Key type lgssc unregistered [ 3330.654910] LNet: 144796:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3330.665930] LNetError: 144796:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3330.686920] LNet: Removed LNI 192.168.206.121@tcp [ 3331.710585] Key type .llcrypt unregistered [ 3331.716404] Key type ._llcrypt unregistered [ 3383.651590] Key type ._llcrypt registered [ 3383.653218] Key type .llcrypt registered [ 3383.846962] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 3400.340092] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 3401.386596] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3401.559577] alg: No test for adler32 (adler32-zlib) [ 3402.822198] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 3403.142983] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 3404.919307] Key type lgssc registered [ 3406.737247] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3450.074814] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3450.106938] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3451.608680] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3451.784588] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3451.862035] Lustre: lustre-MDT0000: new disk, initializing [ 3452.073429] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3452.122692] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3456.880438] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3469.048181] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3469.246100] Lustre: 148970:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3469.323786] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3469.326766] Lustre: Skipped 1 previous similar message [ 3469.412600] Lustre: lustre-MDT0001: new disk, initializing [ 3469.543687] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3469.601759] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3469.617978] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3475.031334] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3485.788804] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3491.593692] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3498.696056] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3498.941505] Lustre: lustre-OST0000: new disk, initializing [ 3498.945093] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3498.951687] Lustre: 150939:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3499.110344] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3504.462406] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3505.235635] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3505.256268] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3505.386812] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3513.535639] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3517.803995] Lustre: Failing over lustre-OST0000 [ 3517.996571] Lustre: server umount lustre-OST0000 complete [ 3535.371838] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3535.605489] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3535.628752] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3535.643962] Lustre: lustre-OST0000: in recovery but waiting for the first client to connect [ 3535.745562] Lustre: lustre-OST0000: Will be in recovery for at least 1:00, or until 3 clients reconnect [ 3536.352488] Lustre: 147191:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788794371/real 1788794371] req@00000000463f74ab x1875686723716864/t0(0) o6->lustre-OST0000-osc-MDT0000@0@lo:28/4 lens 544/432 e 0 to 1 dl 1788794387 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3537.377254] Lustre: 147190:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788794372/real 1788794372] req@00000000c60f2f6f x1875686723716992/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788794388 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3537.401776] Lustre: 147190:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3541.031267] Lustre: lustre-OST0000: Recovery over after 0:06, of 3 clients 3 recovered and 0 were evicted. [ 3541.031402] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3541.554553] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3542.047205] Lustre: 147190:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788794377/real 1788794377] req@000000004b12c37d x1875686723718912/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788794393 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3542.074790] Lustre: 147190:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3545.058508] Lustre: Failing over lustre-MDT0000 [ 3545.424631] Lustre: server umount lustre-MDT0000 complete [ 3546.082215] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3546.094756] Lustre: Skipped 3 previous similar messages [ 3547.106286] Lustre: 147190:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788794382/real 1788794382] req@00000000745a27ba x1875686723720064/t0(0) o400->lustre-OST0000-osc-MDT0001@0@lo:28/4 lens 224/224 e 0 to 1 dl 1788794398 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3547.135970] Lustre: 147190:0:(client.c:2503:ptlrpc_expire_one_request()) Skipped 1 previous similar message [ 3551.093266] LustreError: 148977:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3551.125920] LustreError: 148977:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [ 3556.206528] LustreError: 148979:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3556.224883] LustreError: 148979:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3561.334948] LustreError: 148978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3561.360848] LustreError: 148978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3562.464352] Lustre: 147192:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788794397/real 1788794397] req@00000000da790972 x1875686723728640/t0(0) o400->MGC192.168.206.121@tcp@0@lo:26/25 lens 224/224 e 0 to 1 dl 1788794413 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3562.495737] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3562.916573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3566.443512] LustreError: 148978:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3566.463479] LustreError: 148978:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 3571.562662] LustreError: 151340:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3571.576611] LustreError: 151340:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 3572.707903] LustreError: 147189:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@0000000078e623e0 x1875686723734656/t0(0) o250->MGC192.168.206.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 3572.959704] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3573.008450] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 3576.621431] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3577.717846] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 3578.346399] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 3578.352520] Lustre: Skipped 1 previous similar message [ 3578.368801] Lustre: lustre-MDT0000: Recovery over after 0:01, of 2 clients 2 recovered and 0 were evicted. [ 3578.401247] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 3583.970087] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 3583.978674] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3583.990223] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 3585.808164] Lustre: server umount lustre-OST0000 complete [ 3593.701275] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 3593.706146] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 3593.711279] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 3593.720908] Lustre: Skipped 1 previous similar message [ 3595.615194] Lustre: server umount lustre-MDT0000 complete [ 3598.764763] LustreError: 148962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788794449 with bad export cookie 1252621274989956909 [ 3598.768349] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 3598.776870] LustreError: 148962:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 3608.191791] Lustre: server umount lustre-MDT0001 complete [ 3616.672343] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 3618.720609] Key type lgssc unregistered [ 3618.926562] LNet: 154604:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 3618.936504] LNetError: 154604:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 3618.958305] LNet: Removed LNI 192.168.206.121@tcp [ 3619.570368] Key type .llcrypt unregistered [ 3619.575837] Key type ._llcrypt unregistered [ 3651.853873] Key type ._llcrypt registered [ 3651.855994] Key type .llcrypt registered [ 3651.971368] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 3652.858538] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 3652.909955] alg: No test for adler32 (adler32-zlib) [ 3653.993886] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 3654.271067] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 3655.991145] Key type lgssc registered [ 3657.057630] Lustre: Echo OBD driver; http://www.lustre.org/ [ 3663.331148] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 3663.345451] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3664.750961] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3668.361610] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3674.089924] LustreError: 156125:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 3674.108184] LustreError: 156125:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 3675.834771] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3679.028248] LustreError: 156154:0:(osp_object.c:618:osp_attr_get()) lustre-MDT0001-osp-MDT0000: osp_attr_get update error [0x200000009:0x1:0x0]: rc = -5 [ 3679.046626] LustreError: 156154:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -5 [ 3679.059655] LustreError: 156154:0:(lod_dev.c:511:lod_sub_recovery_thread()) lustre-MDT0001-osp-MDT0000: get update log duration 15, retries 0, failed: rc = -5 [ 3679.494516] Lustre: server umount lustre-MDT0000 complete [ 3692.412920] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 3700.547585] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 3732.964180] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3733.257622] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 3733.275789] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 3733.363673] Lustre: lustre-MDT0000: new disk, initializing [ 3733.447036] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 3733.465671] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 3737.682735] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3748.289340] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 3748.404348] Lustre: 159457:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 3748.449607] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 3748.455505] Lustre: Skipped 1 previous similar message [ 3748.582448] Lustre: lustre-MDT0001: new disk, initializing [ 3748.713323] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 3748.750950] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 3748.760831] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 3752.502673] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3760.353387] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 3764.223894] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 3769.461457] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 3769.630421] Lustre: lustre-OST0000: new disk, initializing [ 3769.633653] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 3769.638367] Lustre: 161424:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 3769.720163] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 3774.835959] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 3778.649320] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 3778.661960] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 3778.769392] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 3782.430595] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 3786.552947] Lustre: DEBUG MARKER: SKIP: conf-sanity test_47b Can not create symlink /usr/sbin/lctl [ 3788.024369] Lustre: DEBUG MARKER: == conf-sanity test 48: too many acls on file ============ 11:23:58 (1788794638) [ 4014.560506] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4014.561542] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4014.571337] Lustre: Skipped 2 previous similar messages [ 4014.588606] Lustre: Skipped 2 previous similar messages [ 4019.685330] LustreError: 159462:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4019.711632] LustreError: 159462:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4019.711875] Lustre: server umount lustre-MDT0000 complete [ 4023.860926] LustreError: 159449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788794874 with bad export cookie 219501580192290396 [ 4023.870484] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4023.875338] LustreError: 159449:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4024.803394] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4024.805513] LustreError: 160704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4024.820869] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4028.831780] LustreError: 159464:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4028.833789] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 4030.177774] Lustre: server umount lustre-MDT0001 complete [ 4039.856672] Lustre: server umount lustre-OST0000 complete [ 4048.298423] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 4057.593557] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4087.825101] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4088.098104] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4088.136376] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4088.253580] Lustre: lustre-MDT0000: new disk, initializing [ 4088.395131] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4088.422050] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4093.140752] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4103.753078] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4103.890371] Lustre: 166076:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4103.940144] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 4103.943670] Lustre: Skipped 1 previous similar message [ 4103.999785] Lustre: lustre-MDT0001: new disk, initializing [ 4104.068303] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4104.117666] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 4104.140516] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4108.016495] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4115.883188] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4120.276795] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4126.049873] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4126.292523] Lustre: lustre-OST0000: new disk, initializing [ 4126.298234] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4126.303485] Lustre: 168046:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4126.371330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4131.799619] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4132.419125] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4132.434455] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4132.552656] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4140.386142] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4144.134636] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4144.379731] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 4148.302632] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4148.466427] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 4153.316063] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4153.317203] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4153.334447] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4153.336628] LustreError: Skipped 1 previous similar message [ 4153.355064] Lustre: Skipped 1 previous similar message [ 4156.514634] Lustre: server umount lustre-OST0000 complete [ 4160.480425] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4160.493267] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4160.505557] Lustre: Skipped 1 previous similar message [ 4160.510282] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4168.672973] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4168.687903] Lustre: Skipped 3 previous similar messages [ 4174.303335] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4174.409216] Lustre: server umount lustre-MDT0000 complete [ 4176.940220] LustreError: 166069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788795027 with bad export cookie 219501580192449233 [ 4176.943801] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4176.950633] LustreError: 166069:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 4177.138244] Lustre: server umount lustre-MDT0001 complete [ 4182.706619] Lustre: DEBUG MARKER: == conf-sanity test 49a: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:30:33 (1788795033) [ 4195.174490] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 4202.576487] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4234.885266] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4235.123264] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4235.147323] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4235.243374] Lustre: lustre-MDT0000: new disk, initializing [ 4235.294904] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4235.305416] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4239.361660] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4248.676681] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4248.890354] Lustre: 172803:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4249.227199] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4253.007804] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4259.957645] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4264.176862] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4269.624438] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4269.887481] Lustre: lustre-OST0000: new disk, initializing [ 4269.890146] Lustre: Skipped 1 previous similar message [ 4269.893478] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 4269.903261] Lustre: Skipped 2 previous similar messages [ 4269.916511] Lustre: 174779:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4275.138279] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4277.839989] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 4277.850333] Lustre: Skipped 1 previous similar message [ 4277.856328] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 4277.940953] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4282.969226] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4288.485763] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4288.486422] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4288.492681] LustreError: Skipped 1 previous similar message [ 4288.502372] Lustre: Skipped 1 previous similar message [ 4288.506855] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4288.516700] Lustre: Skipped 3 previous similar messages [ 4294.293802] Lustre: server umount lustre-OST0000 complete [ 4298.721632] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4298.742143] Lustre: Skipped 2 previous similar messages [ 4303.937421] Lustre: server umount lustre-MDT0000 complete [ 4307.742659] LustreError: 172795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788795158 with bad export cookie 219501580192450108 [ 4307.754104] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4307.754612] LustreError: 172795:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 4308.001833] Lustre: server umount lustre-MDT0001 complete [ 4315.934227] Lustre: DEBUG MARKER: == conf-sanity test 49b: check PARAM_SYS_LDLM_TIMEOUT option of mkfs.lustre ========================================================== 11:32:46 (1788795166) [ 4333.158630] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 4341.593684] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4373.919667] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4374.104812] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4374.134631] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 4374.218442] Lustre: lustre-MDT0000: new disk, initializing [ 4374.296488] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4374.301348] Lustre: Skipped 2 previous similar messages [ 4374.316923] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 4378.433350] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4387.311177] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4387.431283] Lustre: 179101:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4387.719596] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 4391.137964] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4399.013649] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4404.269904] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4411.220218] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4411.419759] Lustre: 181076:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4417.167662] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4417.709417] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 4425.400749] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4432.866469] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4432.870956] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4432.884479] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4432.890073] Lustre: Skipped 6 previous similar messages [ 4433.082975] Lustre: server umount lustre-OST0000 complete [ 4451.295265] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 4451.617782] Lustre: server umount lustre-MDT0000 complete [ 4453.857158] LustreError: 179108:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4453.875046] LustreError: 179108:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4455.352038] LustreError: 179093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788795306 with bad export cookie 219501580192451200 [ 4455.352868] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4455.371919] LustreError: 179093:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 4455.653476] Lustre: server umount lustre-MDT0001 complete [ 4464.526118] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 4466.904540] Key type lgssc unregistered [ 4467.176823] LNet: 182937:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4467.186030] LNetError: 182937:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4468.198786] LNet: Removed LNI 192.168.206.121@tcp [ 4468.855969] Key type .llcrypt unregistered [ 4468.858181] Key type ._llcrypt unregistered [ 4487.809536] Key type ._llcrypt registered [ 4487.812043] Key type .llcrypt registered [ 4487.925785] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4488.795045] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4488.825305] alg: No test for adler32 (adler32-zlib) [ 4489.862390] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 4490.061970] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 4491.711159] Key type lgssc registered [ 4492.583724] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4498.789962] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4498.808475] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4500.165436] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4503.884408] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4510.691508] LustreError: 184335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4510.707631] LustreError: 184335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4511.481217] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4511.967029] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4517.052498] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4526.253640] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4531.965935] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4538.151115] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4538.476771] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4542.758314] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4545.651250] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:3 to 0x280000401:33) [ 4548.485701] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4576.740396] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 4576.746367] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4576.767363] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4577.760811] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4577.778513] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4577.783036] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4577.811177] Lustre: Skipped 1 previous similar message [ 4580.791135] Lustre: server umount lustre-OST0000 complete [ 4588.001841] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4588.012224] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4588.026561] Lustre: Skipped 1 previous similar message [ 4591.765474] Lustre: server umount lustre-MDT0000 complete [ 4593.130196] LustreError: 184335:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4593.183038] LustreError: 184335:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4596.424842] LustreError: 184315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788795447 with bad export cookie 11972806938893455469 [ 4596.433957] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4596.436382] LustreError: 184315:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 4596.780130] Lustre: server umount lustre-MDT0001 complete [ 4606.223038] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 4609.167407] Key type lgssc unregistered [ 4609.428841] LNet: 187893:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 4609.443813] LNetError: 187893:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 4609.468265] LNet: Removed LNI 192.168.206.121@tcp [ 4610.277780] Key type .llcrypt unregistered [ 4610.283277] Key type ._llcrypt unregistered [ 4633.823475] Key type ._llcrypt registered [ 4633.825527] Key type .llcrypt registered [ 4633.919635] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 4635.146681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 4635.416034] alg: No test for adler32 (adler32-zlib) [ 4636.654853] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 4636.840805] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 4638.623171] Key type lgssc registered [ 4640.066232] Lustre: Echo OBD driver; http://www.lustre.org/ [ 4649.104078] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 4649.187304] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4650.940658] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4656.556877] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4660.198109] LustreError: 189311:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4660.215937] LustreError: 189311:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 4664.117568] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4664.413252] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4668.266812] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4674.592614] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4679.501296] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4684.658806] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4684.943717] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4689.622647] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4695.215152] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 4696.680819] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4704.656177] Lustre: lustre-OST0000: Not available for connect from 192.168.206.21@tcp (stopping) [ 4704.742159] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4704.756353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4705.761250] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4705.762130] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4705.774606] Lustre: Skipped 2 previous similar messages [ 4709.409717] Lustre: server umount lustre-OST0000 complete [ 4716.441777] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state DISCONN osc.lustre-OST0000-osc-ffff8af9476a4800.ost_server_uuid 50 [ 4718.022717] Lustre: DEBUG MARKER: OSCs should all be DISCONN [ 4735.458485] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4735.462513] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 4735.481212] Lustre: Skipped 1 previous similar message [ 4735.502696] Lustre: Skipped 1 previous similar message [ 4737.511742] Lustre: server umount lustre-MDT0000 complete [ 4740.576599] LustreError: 190039:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4740.594457] LustreError: 190039:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 4742.289884] LustreError: 189290:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788795593 with bad export cookie 13449411587381981814 [ 4742.325039] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4742.850534] Lustre: server umount lustre-MDT0001 complete [ 4755.305589] Lustre: DEBUG MARKER: == conf-sanity test 50c: lazystatfs one server down ====== 11:40:05 (1788795605) [ 4761.885573] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4762.213378] LustreError: 193228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4762.264758] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4766.434553] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4767.734613] LustreError: 193227:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4772.835061] LustreError: 193228:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4774.047459] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4774.614851] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 4780.120471] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4787.433409] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4792.636067] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4799.414217] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4799.898730] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 4807.114519] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 4807.348315] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4815.359643] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4820.746251] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4820.857263] Lustre: lustre-OST0001: new disk, initializing [ 4820.860115] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 4820.864293] Lustre: 196068:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 4820.934887] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 4825.671403] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4826.191124] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 4826.196429] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 4826.287082] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 4833.645357] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4836.836488] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4836.848311] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4836.857090] Lustre: Skipped 1 previous similar message [ 4836.871318] Lustre: Skipped 1 previous similar message [ 4841.960415] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4841.969748] Lustre: Skipped 2 previous similar messages [ 4842.122446] Lustre: server umount lustre-OST0000 complete [ 4844.399394] LustreError: 195048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4847.926352] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 4848.073595] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 4851.956693] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 4852.147762] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 4862.437694] LustreError: 195048:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4862.446805] LustreError: 195048:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 10 previous similar messages [ 4867.560962] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4867.563252] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4867.584754] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 4867.592947] Lustre: Skipped 1 previous similar message [ 4872.052244] Lustre: server umount lustre-OST0001 complete [ 4876.782697] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 4876.797630] LustreError: Skipped 1 previous similar message [ 4876.805468] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4882.877711] Lustre: server umount lustre-MDT0000 complete [ 4886.477223] LustreError: 196067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788795737 with bad export cookie 13449411587381982885 [ 4886.480138] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 4886.486456] LustreError: 196067:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 4886.720535] Lustre: server umount lustre-MDT0001 complete [ 4905.831513] Lustre: DEBUG MARKER: == conf-sanity test 50d: lazystatfs client/server conn race ========================================================== 11:42:36 (1788795756) [ 4911.056429] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4911.293350] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 4911.341504] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 4911.509530] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 4915.067203] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4921.116777] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 4921.182497] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 4921.207123] Lustre: 199502:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 4924.461618] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4929.876743] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 4934.188240] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 4940.137316] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4940.331190] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 4945.998487] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 4946.472422] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4954.503200] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4960.757407] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 4960.838090] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 4966.350261] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 4972.515973] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 4976.609131] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4976.619106] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 4976.628842] Lustre: Skipped 1 previous similar message [ 4976.634730] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 4976.641570] Lustre: Skipped 4 previous similar messages [ 4981.305813] Lustre: server umount lustre-OST0000 complete [ 4983.164400] LustreError: 201352:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 4983.186043] LustreError: 201352:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 4998.630925] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 4998.638594] LustreError: Skipped 1 previous similar message [ 5001.764754] Lustre: server umount lustre-OST0001 complete [ 5008.352727] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 5008.361598] LustreError: Skipped 1 previous similar message [ 5009.378112] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 5009.381035] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5009.394916] Lustre: Skipped 8 previous similar messages [ 5009.414929] Lustre: Skipped 4 previous similar messages [ 5011.639337] Lustre: server umount lustre-MDT0000 complete [ 5015.401863] LustreError: 199495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788795866 with bad export cookie 13449411587381984292 [ 5015.403902] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5015.416484] LustreError: 199495:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5015.804809] Lustre: server umount lustre-MDT0001 complete [ 5037.469408] Lustre: DEBUG MARKER: == conf-sanity test 50e: normal statfs all servers down == 11:44:48 (1788795888) [ 5050.634720] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 5059.005344] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5088.157847] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5088.372944] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5088.404603] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 5088.476285] Lustre: lustre-MDT0000: new disk, initializing [ 5088.540981] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5088.546130] Lustre: Skipped 3 previous similar messages [ 5088.558674] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 5092.420566] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5101.996695] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5102.068501] Lustre: 207598:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5102.117698] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 5102.125922] Lustre: Skipped 1 previous similar message [ 5102.237798] Lustre: lustre-MDT0001: new disk, initializing [ 5102.368247] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 5102.380653] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 5105.897702] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5113.788649] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5118.648990] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5123.773781] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5124.031283] Lustre: lustre-OST0000: new disk, initializing [ 5124.038094] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 5124.044285] Lustre: 209574:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5129.807236] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5131.383442] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 5131.409985] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 5131.540096] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 5141.504533] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5146.166976] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5146.339913] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5150.854473] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5151.005832] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5157.349880] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 5157.356353] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5157.357111] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5157.377117] Lustre: Skipped 1 previous similar message [ 5157.381312] Lustre: Skipped 1 previous similar message [ 5159.038366] Lustre: server umount lustre-OST0000 complete [ 5169.292087] Lustre: server umount lustre-MDT0000 complete [ 5172.706661] LustreError: 207605:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5172.732734] LustreError: 207605:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 13 previous similar messages [ 5173.103898] LustreError: 207591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788796024 with bad export cookie 13449411587381985853 [ 5173.107721] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5173.115763] LustreError: 207591:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [ 5173.562508] Lustre: server umount lustre-MDT0001 complete [ 5180.390293] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5184.802050] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5193.919255] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5198.903760] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5208.058504] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5213.517084] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5219.570916] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5219.927196] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5219.935137] Lustre: Skipped 4 previous similar messages [ 5225.261569] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5233.252541] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5237.373312] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5237.600271] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5243.085502] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5243.319740] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5245.924913] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5245.939329] LustreError: Skipped 2 previous similar messages [ 5251.665306] Lustre: server umount lustre-OST0000 complete [ 5257.017392] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5261.211571] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5289.690204] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5295.793678] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5301.357465] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.380181] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 5301.385461] Lustre: lustre-OST0000-osc-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5301.401403] Lustre: Skipped 1 previous similar message [ 5302.763438] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5309.520466] Lustre: DEBUG MARKER: osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 5 sec [ 5317.091696] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5317.093170] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5317.111823] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 5317.112774] Lustre: Skipped 5 previous similar messages [ 5317.125458] Lustre: Skipped 9 previous similar messages [ 5327.328044] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5327.502552] Lustre: server umount lustre-OST0000 complete [ 5345.247135] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5345.426633] Lustre: server umount lustre-MDT0000 complete [ 5347.816451] LustreError: 211788:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5347.835612] LustreError: 211788:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 5348.406599] LustreError: 211774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788796199 with bad export cookie 13449411587381986728 [ 5348.411989] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5348.413974] LustreError: 211774:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5348.676317] Lustre: server umount lustre-MDT0001 complete [ 5355.738679] Lustre: DEBUG MARKER: == conf-sanity test 50f: normal statfs one server in down ========================================================== 11:50:06 (1788796206) [ 5361.635058] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5365.721106] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5373.933623] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5378.858662] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5385.712483] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5390.203371] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5395.802524] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5401.564737] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5408.496215] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5412.248235] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 5412.447080] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5416.808544] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 5416.993837] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5423.627173] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5423.870881] Lustre: lustre-OST0001: new disk, initializing [ 5423.874421] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 5423.879653] Lustre: 221129:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 5431.093929] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5431.965415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 5431.977084] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 5432.198553] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 5440.805780] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5445.871634] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5446.089786] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5450.055905] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5450.214622] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5452.779652] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5452.787032] LustreError: Skipped 2 previous similar messages [ 5466.080640] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5466.188650] Lustre: server umount lustre-OST0001 complete [ 5471.294176] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5471.448655] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in DISCONN state after 0 sec [ 5475.756512] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state DISCONN os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5476.044874] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in DISCONN state after 0 sec [ 5505.800462] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5506.076055] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5506.079844] Lustre: Skipped 5 previous similar messages [ 5511.162816] LustreError: lustre-OST0001-osc-MDT0001: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5511.170894] LustreError: lustre-OST0001-osc-MDT0000: This client was evicted by lustre-OST0001; in progress operations using this service will fail. [ 5511.171756] Lustre: lustre-OST0001-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 5511.200062] Lustre: Skipped 1 previous similar message [ 5511.900065] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5518.639301] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5520.807582] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5537.247452] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 5537.585914] Lustre: server umount lustre-OST0001 complete [ 5562.125514] Lustre: server umount lustre-OST0000 complete [ 5572.951119] Lustre: server umount lustre-MDT0000 complete [ 5576.870335] LustreError: 217742:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788796427 with bad export cookie 13449411587381987701 [ 5576.889693] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5583.214381] Lustre: server umount lustre-MDT0001 complete [ 5605.965374] Lustre: DEBUG MARKER: == conf-sanity test 50g: deactivated OST should not cause panic ========================================================== 11:54:16 (1788796456) [ 5611.535454] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5611.820171] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5611.846275] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5616.090161] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5626.780720] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5626.895388] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5626.938631] Lustre: 226376:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5631.243450] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5641.069481] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5650.066450] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5659.674544] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5659.969403] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5666.551648] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5673.834686] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5680.879408] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5680.941650] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5686.827301] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5693.438740] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5695.023631] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 5698.652987] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid 50 [ 5698.832175] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 5702.387343] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid 50 [ 5702.583202] Lustre: DEBUG MARKER: os[cp].lustre-OST0001-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 5707.781497] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state (FULL|IDLE) osc.lustre-OST0001-osc-ffff8af9505c5000.ost_server_uuid 50 [ 5709.879833] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-ffff8af9505c5000.ost_server_uuid in IDLE state after 0 sec [ 5710.595979] Lustre: Permanently deactivating lustre-OST0001 [ 5710.612300] Lustre: Setting parameter lustre-OST0001-osc.osc.active=0 in log lustre-client [ 5713.930852] Lustre: Permanently reactivating lustre-OST0001 [ 5722.592871] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 5722.600719] LustreError: Skipped 3 previous similar messages [ 5722.607676] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5722.616310] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 5722.621961] Lustre: Skipped 11 previous similar messages [ 5722.625094] Lustre: Skipped 29 previous similar messages [ 5724.655238] Lustre: server umount lustre-OST0001 complete [ 5727.719087] LustreError: 228226:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5727.746294] LustreError: 228226:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 35 previous similar messages [ 5734.098471] Lustre: server umount lustre-OST0000 complete [ 5743.811366] Lustre: server umount lustre-MDT0000 complete [ 5748.545045] LustreError: 227130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788796599 with bad export cookie 13449411587381988989 [ 5748.554619] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5748.563638] LustreError: 227130:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 5748.973522] Lustre: server umount lustre-MDT0001 complete [ 5772.203501] Lustre: DEBUG MARKER: == conf-sanity test 50h: LU-642: activate deactivated OST ========================================================== 11:57:02 (1788796622) [ 5779.634901] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5779.948517] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 5779.995304] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 5780.004803] Lustre: Skipped 5 previous similar messages [ 5783.857137] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5790.594307] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5790.656460] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 5790.677955] Lustre: 233071:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5794.391131] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5800.549518] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5805.326349] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5810.889638] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5811.079918] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 5811.106811] Lustre: Permanently deactivating lustre-OST0000 [ 5811.125798] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [ 5815.902919] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5817.371284] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [ 5822.129745] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5828.026763] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5828.093391] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 5832.822931] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5836.954765] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:65) [ 5838.671914] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5839.890269] Lustre: Permanently reactivating lustre-OST0000 [ 5839.909193] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [ 5839.914678] Lustre: Skipped 2 previous similar messages [ 5843.872063] Lustre: lustre-OST0000: Received new MDS connection from 0@lo, remove former export from same NID [ 5843.940505] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 5854.222842] Lustre: server umount lustre-OST0001 complete [ 5863.302664] Lustre: server umount lustre-OST0000 complete [ 5872.411833] Lustre: server umount lustre-MDT0000 complete [ 5875.164512] LustreError: 233811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788796726 with bad export cookie 13449411587381990634 [ 5875.171630] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 5875.175764] LustreError: 233811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 5875.469162] Lustre: server umount lustre-MDT0001 complete [ 5884.382788] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 5886.609946] Key type lgssc unregistered [ 5886.829823] LNet: 237948:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 5886.833830] LNetError: 237948:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 5886.845274] LNet: Removed LNI 192.168.206.121@tcp [ 5887.585750] Key type .llcrypt unregistered [ 5887.587431] Key type ._llcrypt unregistered [ 5906.437087] Key type ._llcrypt registered [ 5906.443141] Key type .llcrypt registered [ 5906.560953] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 5907.360261] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 5907.389239] alg: No test for adler32 (adler32-zlib) [ 5908.416402] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 5908.616122] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 5910.297785] Key type lgssc registered [ 5911.401299] Lustre: Echo OBD driver; http://www.lustre.org/ [ 5920.248319] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 5920.267598] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5921.561051] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 5925.204583] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5930.948793] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 5930.978477] LustreError: 239398:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5930.993786] LustreError: 239398:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 5931.070101] Lustre: Found index 1 for lustre-MDT0001, updating log [ 5931.114614] Lustre: 239384:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 5931.129793] Lustre: Permanently deactivating lustre-MDT0001 [ 5931.141330] Lustre: Setting parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 5931.407101] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 5934.636676] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5936.659464] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 5939.882148] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 5943.890696] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5948.573663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5948.858412] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 5953.523516] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5956.009401] LustreError: 241241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5956.018545] Lustre: lustre-OST0000: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x280000400:131 to 0x280000400:161) [ 5961.032537] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5961.193992] LustreError: 241239:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.310671] LustreError: 241241:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 5966.330182] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:161) [ 5966.332361] LustreError: 241241:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 5968.126788] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 5968.317363] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 5973.483505] Lustre: lustre-OST0001: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x2c0000401:35 to 0x2c0000401:97) [ 5973.484797] Lustre: lustre-OST0001: new connection from lustre-MDT0001-mdtlov (cleaning up unused objects from 0x2c0000400:67 to 0x2c0000400:97) [ 5974.663629] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 5984.721321] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 5988.938993] Lustre: Permanently reactivating lustre-MDT0001 [ 5988.956125] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=1 in log lustre-client [ 5988.968294] Lustre: Skipped 2 previous similar messages [ 5990.698817] Lustre: lustre-MDT0001: Client 9e93c9b4-88b4-4162-b22d-7280bd90b60a (at 192.168.206.21@tcp) reconnecting [ 5996.651830] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 5996.673180] Lustre: lustre-MDT0001: Received new MDS connection from 0@lo, keep former export from same NID [ 5996.686850] LustreError: lustre-MDT0001-osp-MDT0000: This client was evicted by lustre-MDT0001; in progress operations using this service will fail. [ 5996.719520] LustreError: 243117:0:(llog_osd.c:2268:llog_osd_get_cat_list()) lustre-MDT0001-osp-MDT0000: error reading CATALOGS: rc = -108 [ 5996.720497] Lustre: lustre-MDT0001-osp-MDT0000: Connection restored to 0@lo (at 0@lo) [ 5996.741498] LustreError: 243117:0:(lod_sub_object.c:917:lod_sub_prep_llog()) lustre-MDT0000-mdtlov: can't get id from catalogs: rc = -108 [ 5996.743348] LustreError: 243117:0:(obd_config.c:2102:class_config_llog_handler()) MGC192.168.206.121@tcp: cfg command failed: rc = -108 [ 5996.743392] Lustre: cmd=cf00f 0:lustre-MDT0000-mdtlov 1:lustre-MDT0001-osp-MDT0000.active=1 [ 5996.743392] [ 5996.745293] LustreError: 239389:0:(mgc_request.c:614:do_requeue()) failed processing log: -108 [ 5997.238431] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 5998.850457] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 6004.069502] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [ 6004.323632] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [ 6009.035806] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [ 6009.324485] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [ 6012.237052] Lustre: Permanently deactivating lustre-MDT0001 [ 6012.252644] Lustre: Modifying parameter lustre-MDT0001-mdc.mdc.active=0 in log lustre-client [ 6012.256211] Lustre: Skipped 2 previous similar messages [ 6019.114345] Lustre: setting import lustre-MDT0001_UUID INACTIVE by administrator request [ 6028.768594] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6028.773951] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6028.788929] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6032.864710] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6032.869728] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6032.877484] Lustre: Skipped 2 previous similar messages [ 6032.906267] Lustre: Skipped 2 previous similar messages [ 6034.503809] Lustre: server umount lustre-MDT0000 complete [ 6037.739369] LustreError: 239379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788796888 with bad export cookie 332820532371194331 [ 6037.749229] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6037.752151] LustreError: 239379:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6037.989978] LustreError: 239393:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6038.015246] LustreError: 239393:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6038.143852] Lustre: server umount lustre-MDT0001 complete [ 6048.289210] Lustre: server umount lustre-OST0000 complete [ 6059.903338] Lustre: DEBUG MARKER: == conf-sanity test 51: Verify that mdt_reint handles RMF_MDT_MD correctly when an OST is added ========================================================== 12:01:50 (1788796910) [ 6069.734687] Lustre: server umount lustre-OST0001 complete [ 6075.798994] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 6084.478314] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 6125.354837] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6126.173633] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6126.194707] Lustre: Skipped 2 previous similar messages [ 6126.264451] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 6126.443491] Lustre: lustre-MDT0000: new disk, initializing [ 6126.634865] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6126.682702] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 6132.779551] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6145.200635] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6145.374982] Lustre: 247686:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6145.447696] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 6145.452200] Lustre: Skipped 1 previous similar message [ 6145.615239] Lustre: lustre-MDT0001: new disk, initializing [ 6145.741187] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6145.818698] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 6145.843850] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 6150.916976] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6161.476390] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6167.646782] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6175.109386] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6175.340365] Lustre: lustre-OST0000: new disk, initializing [ 6175.343826] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 6175.348879] Lustre: 249657:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6175.414105] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6181.934077] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 6181.941369] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 6182.052367] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 6182.057391] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6191.046338] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6193.373864] LustreError: 249250:0:(mdt_handler.c:3309:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6203.065950] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6203.262917] Lustre: lustre-OST0001: new disk, initializing [ 6203.266936] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 6203.273923] Lustre: 250771:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 6203.403868] LustreError: 249250:0:(mdt_handler.c:3309:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6203.423630] LustreError: 247694:0:(mdt_handler.c:3309:mdt_reint_internal()) cfs_fail_timeout id 142 sleeping for 10000ms [ 6209.930530] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6210.654456] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 6210.660595] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 6210.783067] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 6213.481467] LustreError: 247694:0:(mdt_handler.c:3309:mdt_reint_internal()) cfs_fail_timeout id 142 awake [ 6219.461334] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6220.894757] Lustre: DEBUG MARKER: osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid in FULL state after 0 sec [ 6226.423590] LustreError: lustre-OST0001-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6226.425443] Lustre: lustre-OST0001-osc-MDT0000: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6226.431874] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6226.448056] Lustre: Skipped 1 previous similar message [ 6226.452302] Lustre: Skipped 1 previous similar message [ 6231.533571] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6231.543391] Lustre: Skipped 1 previous similar message [ 6236.654534] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 6236.664026] Lustre: Skipped 1 previous similar message [ 6237.152424] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6237.303954] Lustre: server umount lustre-OST0001 complete [ 6241.763693] LustreError: 249639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6241.788809] LustreError: 249639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6254.050576] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6254.059820] LustreError: Skipped 1 previous similar message [ 6254.064347] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6254.089591] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6262.266811] LustreError: 249639:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6262.283504] LustreError: 249639:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 6268.383425] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6268.486247] Lustre: server umount lustre-OST0000 complete [ 6272.482138] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6272.512159] Lustre: Skipped 1 previous similar message [ 6272.521798] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6272.533903] Lustre: Skipped 7 previous similar messages [ 6278.563294] Lustre: server umount lustre-MDT0000 complete [ 6282.432382] LustreError: 247677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797133 with bad export cookie 332820532371197145 [ 6282.455406] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6282.474512] LustreError: 247677:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6288.915871] Lustre: server umount lustre-MDT0001 complete [ 6297.963762] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 6300.192378] Key type lgssc unregistered [ 6300.441626] LNet: 252816:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6300.447449] LNetError: 252816:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6300.469140] LNet: Removed LNI 192.168.206.121@tcp [ 6301.196520] Key type .llcrypt unregistered [ 6301.198925] Key type ._llcrypt unregistered [ 6339.850891] Key type ._llcrypt registered [ 6339.853202] Key type .llcrypt registered [ 6339.972648] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 6341.128526] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6341.170943] alg: No test for adler32 (adler32-zlib) [ 6342.172141] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 6342.357947] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 6344.023413] Key type lgssc registered [ 6345.150608] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6352.044647] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6352.067418] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6353.334253] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 6353.350705] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 6353.509609] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6357.658840] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6366.625133] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6366.770744] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 6366.813190] Lustre: 255025:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 6367.141658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6371.924958] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6379.322549] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6385.595716] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6392.914521] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6393.067774] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 6393.328330] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6401.085918] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6403.260500] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:4 to 0x280000401:33) [ 6410.542531] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6464.480948] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6464.492580] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6464.506644] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6464.994058] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6465.011103] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6468.701519] Lustre: server umount lustre-OST0000 complete [ 6472.511988] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: (null) [ 6483.410073] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6483.761446] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6485.805302] LustreError: lustre-OST0000-osc-MDT0001: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.820311] LustreError: lustre-OST0000-osc-MDT0000: This client was evicted by lustre-OST0000; in progress operations using this service will fail. [ 6485.821971] Lustre: lustre-OST0000-osc-MDT0001: Connection restored to 0@lo (at 0@lo) [ 6485.839931] Lustre: Skipped 1 previous similar message [ 6489.329527] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6495.583508] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6537.184073] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6537.193417] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6537.207206] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6537.240772] Lustre: Skipped 1 previous similar message [ 6539.492875] Lustre: server umount lustre-OST0000 complete [ 6546.408099] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6546.415844] LustreError: Skipped 1 previous similar message [ 6546.419245] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6546.442955] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6546.446435] Lustre: Skipped 1 previous similar message [ 6549.091935] Lustre: server umount lustre-MDT0000 complete [ 6552.457147] LustreError: 255018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797403 with bad export cookie 16201507097719482272 [ 6552.461875] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6552.468731] LustreError: 255018:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6552.786933] Lustre: server umount lustre-MDT0001 complete [ 6560.782128] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 6563.070751] Key type lgssc unregistered [ 6563.379765] LNet: 260453:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6563.394944] LNetError: 260453:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6563.422529] LNet: Removed LNI 192.168.206.121@tcp [ 6564.171269] Key type .llcrypt unregistered [ 6564.173059] Key type ._llcrypt unregistered [ 6584.934593] Key type ._llcrypt registered [ 6584.937090] Key type .llcrypt registered [ 6585.031501] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 6586.317520] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6586.394825] alg: No test for adler32 (adler32-zlib) [ 6587.428942] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 6587.632533] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 6589.361620] Key type lgssc registered [ 6591.161702] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6599.695846] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6599.720143] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6601.258335] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6605.563802] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6611.424925] LustreError: 261851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6611.433650] LustreError: 261851:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6614.253807] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6614.706901] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6619.431469] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6626.408967] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6631.582146] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6637.380558] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6637.689307] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6643.248466] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6646.930975] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:42 to 0x280000401:65) [ 6650.059399] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6670.816254] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 6670.824911] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6670.836753] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6672.867069] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6672.869919] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6672.871462] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6675.616457] Lustre: server umount lustre-OST0000 complete [ 6681.570726] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 6681.575275] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6681.589965] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6681.593868] Lustre: Skipped 1 previous similar message [ 6684.747026] Lustre: server umount lustre-MDT0000 complete [ 6688.227290] LustreError: 261850:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6688.241587] LustreError: 261850:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6688.549737] LustreError: 261832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797539 with bad export cookie 929470621467847049 [ 6688.554373] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6688.558174] LustreError: 261832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 6688.825543] Lustre: server umount lustre-MDT0001 complete [ 6697.213589] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 6699.719967] Key type lgssc unregistered [ 6699.990767] LNet: 266155:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6699.998822] LNetError: 266155:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6700.012128] LNet: Removed LNI 192.168.206.121@tcp [ 6700.688443] Key type .llcrypt unregistered [ 6700.690986] Key type ._llcrypt unregistered [ 6715.083295] Key type ._llcrypt registered [ 6715.086635] Key type .llcrypt registered [ 6715.172747] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 6715.944681] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6715.992081] alg: No test for adler32 (adler32-zlib) [ 6717.035505] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 6717.225888] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 6718.847171] Key type lgssc registered [ 6719.776943] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6726.436988] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6726.450678] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6727.925207] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6731.617522] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6737.390564] LustreError: 267371:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6737.422104] LustreError: 267371:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6740.001644] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6740.378583] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6744.875836] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6752.865832] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6757.855905] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6764.067185] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6764.663293] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6770.926864] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6772.863188] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:67 to 0x280000401:97) [ 6777.794908] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6788.576346] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6788.583709] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6788.593296] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6788.596868] Lustre: Skipped 1 previous similar message [ 6788.604975] Lustre: Skipped 1 previous similar message [ 6793.697924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6793.711056] Lustre: Skipped 1 previous similar message [ 6794.742578] Lustre: server umount lustre-OST0000 complete [ 6798.820994] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6798.833289] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6798.843293] Lustre: Skipped 1 previous similar message [ 6803.939119] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6803.948034] Lustre: Skipped 1 previous similar message [ 6809.056689] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6809.063093] Lustre: Skipped 1 previous similar message [ 6812.127142] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 6812.274279] Lustre: server umount lustre-MDT0000 complete [ 6814.179232] LustreError: 267366:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6814.189915] LustreError: 267366:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6815.470982] LustreError: 267351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797666 with bad export cookie 8350964554532918924 [ 6815.479088] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6815.484981] LustreError: 267351:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 6815.813849] Lustre: server umount lustre-MDT0001 complete [ 6825.583780] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 6828.377587] Key type lgssc unregistered [ 6828.602075] LNet: 271050:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6828.609321] LNetError: 271050:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6828.624197] LNet: Removed LNI 192.168.206.121@tcp [ 6829.433711] Key type .llcrypt unregistered [ 6829.435826] Key type ._llcrypt unregistered [ 6848.467674] Key type ._llcrypt registered [ 6848.469443] Key type .llcrypt registered [ 6848.559767] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 6849.309248] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6849.334245] alg: No test for adler32 (adler32-zlib) [ 6850.354640] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 6850.515718] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 6852.159166] Key type lgssc registered [ 6853.009530] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6859.247331] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6859.278564] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6860.776995] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6864.641829] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6869.984540] LustreError: 272447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6870.005959] LustreError: 272447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 6871.598990] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6872.089164] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 6876.211901] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6882.455399] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 6886.988416] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 6893.884498] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 6894.368318] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 6900.143766] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 6905.647820] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:99 to 0x280000401:129) [ 6907.067988] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 6931.425026] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 6931.430894] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6931.433872] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 6931.442323] Lustre: Skipped 1 previous similar message [ 6931.456120] Lustre: Skipped 1 previous similar message [ 6936.407630] Lustre: server umount lustre-OST0000 complete [ 6941.672347] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 6941.676229] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 6941.691337] Lustre: Skipped 1 previous similar message [ 6945.895835] Lustre: server umount lustre-MDT0000 complete [ 6946.784474] LustreError: 272447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 6946.808065] LustreError: 272447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 6949.154380] LustreError: 272429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797800 with bad export cookie 13248372005717289239 [ 6949.156930] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 6949.163381] LustreError: 272429:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 6949.397552] Lustre: server umount lustre-MDT0001 complete [ 6959.237594] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 6961.725399] Key type lgssc unregistered [ 6961.985710] LNet: 276756:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 6961.993106] LNetError: 276756:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 6962.030094] LNet: Removed LNI 192.168.206.121@tcp [ 6962.752755] Key type .llcrypt unregistered [ 6962.754952] Key type ._llcrypt unregistered [ 6979.642335] Key type ._llcrypt registered [ 6979.650149] Key type .llcrypt registered [ 6979.776735] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 6980.871351] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 6981.007353] alg: No test for adler32 (adler32-zlib) [ 6982.061470] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 6982.234319] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 6983.975629] Key type lgssc registered [ 6984.851674] Lustre: Echo OBD driver; http://www.lustre.org/ [ 6991.263865] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 6991.279084] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 6992.913696] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 6997.226756] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7002.089034] LustreError: 277992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7002.108660] LustreError: 277992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7004.492658] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7004.808053] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7008.419677] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7015.311058] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7020.925347] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7027.368169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7027.777325] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7033.404570] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7036.993120] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:131 to 0x280000401:161) [ 7039.719648] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7050.212535] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7050.218838] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7050.235424] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7052.256400] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7052.267790] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7052.271485] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7052.313388] Lustre: Skipped 1 previous similar message [ 7055.710579] Lustre: server umount lustre-OST0000 complete [ 7060.962393] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7060.975171] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7060.983810] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7065.427484] Lustre: server umount lustre-MDT0000 complete [ 7067.620043] LustreError: 277973:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7067.635471] LustreError: 277973:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7068.567203] LustreError: 278752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788797919 with bad export cookie 5470230799905459215 [ 7068.577689] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7068.582802] LustreError: 278752:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7068.854444] Lustre: server umount lustre-MDT0001 complete [ 7079.115383] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 7081.560923] Key type lgssc unregistered [ 7081.903848] LNet: 281695:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7081.927170] LNetError: 281695:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7081.966986] LNet: Removed LNI 192.168.206.121@tcp [ 7082.986693] Key type .llcrypt unregistered [ 7082.990341] Key type ._llcrypt unregistered [ 7094.781954] Key type ._llcrypt registered [ 7094.784271] Key type .llcrypt registered [ 7094.880347] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing run_llverdev /dev/mapper/ost1_flakey -p [ 7110.500636] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 7125.142421] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 7125.956714] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7125.967789] alg: No test for adler32 (adler32-zlib) [ 7127.072960] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 7127.349883] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 7129.055185] Key type lgssc registered [ 7130.436556] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7160.942579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7160.973649] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7162.202903] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7162.237097] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7162.307315] Lustre: lustre-MDT0000: new disk, initializing [ 7162.370944] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7162.402245] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7166.344613] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7175.284801] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7175.406329] Lustre: 285252:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7175.460540] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7175.464136] Lustre: Skipped 1 previous similar message [ 7175.527442] Lustre: lustre-MDT0001: new disk, initializing [ 7175.601433] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7175.644747] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7175.653756] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7178.970790] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7187.513467] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7193.483920] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7200.544104] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7200.923522] Lustre: lustre-OST0000: new disk, initializing [ 7200.929599] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7200.935575] Lustre: 287221:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7201.044615] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7208.063640] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7208.099093] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7208.300190] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7208.478486] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7220.969102] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7226.407947] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7226.751163] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7232.299981] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7232.679846] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7239.137153] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7239.138977] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7239.153969] LustreError: Skipped 1 previous similar message [ 7239.155923] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7239.229830] Lustre: Skipped 1 previous similar message [ 7244.263980] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7244.274948] Lustre: Skipped 2 previous similar messages [ 7249.376156] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7249.381512] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7249.404315] Lustre: Skipped 1 previous similar message [ 7249.558871] Lustre: server umount lustre-OST0000 complete [ 7254.501930] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7254.522826] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7254.538149] Lustre: Skipped 1 previous similar message [ 7259.616657] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7259.632784] Lustre: Skipped 1 previous similar message [ 7260.091081] Lustre: server umount lustre-MDT0000 complete [ 7263.966367] LustreError: 287220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798115 with bad export cookie 16114226894371030549 [ 7263.968486] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7263.983253] LustreError: 287220:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7264.527228] Lustre: server umount lustre-MDT0001 complete [ 7273.668328] Lustre: DEBUG MARKER: == conf-sanity test 54b: test llverfs and partial verify of filesystem ========================================================== 12:22:03 (1788798123) [ 7280.595583] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7281.241849] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7285.962606] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7286.243411] LustreError: 289635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7286.272772] LustreError: 289635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 7291.378539] LustreError: 289636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7294.401653] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7294.926245] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7299.471715] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7306.640945] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7312.265873] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7318.401445] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7318.705193] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7325.755536] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7334.087256] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7345.644084] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 7345.665266] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7345.694471] Lustre: Skipped 1 previous similar message [ 7345.704163] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7347.696818] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7347.708927] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7351.255492] Lustre: server umount lustre-OST0000 complete [ 7356.389771] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7356.402748] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7361.223369] Lustre: server umount lustre-MDT0000 complete [ 7363.042771] LustreError: 290362:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 7363.066110] LustreError: 290362:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 7364.804395] LustreError: 289617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798215 with bad export cookie 16114226894371031431 [ 7364.807755] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7364.815319] LustreError: 289617:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 7365.256967] Lustre: server umount lustre-MDT0001 complete [ 7375.675562] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 7378.511886] Key type lgssc unregistered [ 7378.911480] LNet: 293171:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7378.926451] LNetError: 293171:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7378.951044] LNet: Removed LNI 192.168.206.121@tcp [ 7379.763366] Key type .llcrypt unregistered [ 7379.765043] Key type ._llcrypt unregistered [ 7404.594514] Key type ._llcrypt registered [ 7404.598675] Key type .llcrypt registered [ 7404.721142] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 7420.165582] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 7421.547375] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7421.716576] alg: No test for adler32 (adler32-zlib) [ 7422.826751] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 7423.098734] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 7424.775225] Key type lgssc registered [ 7426.059379] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7473.207788] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7473.242614] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7474.474188] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7474.501186] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7474.573924] Lustre: lustre-MDT0000: new disk, initializing [ 7474.624683] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7474.644415] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7478.684705] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7487.946896] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7488.082708] Lustre: 296999:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7488.138282] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7488.141559] Lustre: Skipped 1 previous similar message [ 7488.195864] Lustre: lustre-MDT0001: new disk, initializing [ 7488.275194] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7488.315931] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7488.331633] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7492.395504] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7501.620435] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7508.047519] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7515.249579] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7515.453218] Lustre: lustre-OST2710: new disk, initializing [ 7515.456316] Lustre: srv-lustre-OST2710: No data found on store. Initialize space. [ 7515.461092] Lustre: 298972:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7515.532287] Lustre: lustre-OST2710: Imperative Recovery not enabled, recovery window 60-180 [ 7521.829110] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7523.451577] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:2710:ost [ 7523.472065] Lustre: cli-lustre-OST2710-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:2710:ost] [ 7523.648689] Lustre: lustre-OST2710-osc-MDT0000: update sequence from 0x127100000 to 0x280000401 [ 7532.413516] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST2710-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7540.783716] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7541.070200] Lustre: lustre-OST03e8: new disk, initializing [ 7541.075782] Lustre: srv-lustre-OST03e8: No data found on store. Initialize space. [ 7541.088580] Lustre: 300035:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7541.220335] Lustre: lustre-OST03e8: Imperative Recovery not enabled, recovery window 60-180 [ 7549.302873] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7549.564813] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:3e8:ost [ 7549.576460] Lustre: cli-lustre-OST03e8-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:3e8:ost] [ 7549.668724] Lustre: lustre-OST03e8-osc-MDT0000: update sequence from 0x103e80000 to 0x2c0000401 [ 7557.023930] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST03e8-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7563.660677] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid 50 [ 7563.875678] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7568.085825] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid 50 [ 7568.281276] Lustre: DEBUG MARKER: os[cp].lustre-OST2710-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7572.848530] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid 50 [ 7573.062272] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7577.475872] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid 50 [ 7577.621931] Lustre: DEBUG MARKER: os[cp].lustre-OST03e8-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7584.336291] Lustre: server umount lustre-MDT0000 complete [ 7585.777086] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7585.784673] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7585.802956] Lustre: Skipped 3 previous similar messages [ 7588.026596] LustreError: 296990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798439 with bad export cookie 193392606143747621 [ 7588.039419] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7588.047239] LustreError: 296990:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 7588.407685] Lustre: server umount lustre-MDT0001 complete [ 7601.200575] Lustre: server umount lustre-OST2710 complete [ 7615.028496] Lustre: server umount lustre-OST03e8 complete [ 7632.581872] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 7640.553426] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 7670.848801] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7671.016136] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7671.041301] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 7671.124930] Lustre: lustre-MDT0000: new disk, initializing [ 7671.189316] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7671.201130] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 7675.825555] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7686.776193] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7686.882557] Lustre: 305281:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7686.930790] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 7686.934665] Lustre: Skipped 1 previous similar message [ 7686.998469] Lustre: lustre-MDT0001: new disk, initializing [ 7687.113748] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7687.164426] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 7687.174870] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 7690.902843] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7699.629813] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7705.021591] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7710.754462] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7710.947641] Lustre: lustre-OST0000: new disk, initializing [ 7710.951965] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 7710.958403] Lustre: 307254:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 7711.056715] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 7716.392822] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7720.469519] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 7720.477557] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 7720.558407] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 7723.716218] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 7727.794274] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 7727.984988] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 7732.042641] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 7732.204228] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 7735.785159] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7735.810500] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 7735.817853] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7740.913611] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7740.925523] Lustre: Skipped 1 previous similar message [ 7746.023073] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 7746.030613] Lustre: Skipped 1 previous similar message [ 7748.063294] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 7748.224234] Lustre: server umount lustre-OST0000 complete [ 7753.697212] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 7753.710368] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7753.724155] Lustre: Skipped 1 previous similar message [ 7753.729705] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7753.736952] Lustre: Skipped 1 previous similar message [ 7756.258720] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7757.900279] Lustre: server umount lustre-MDT0000 complete [ 7761.171869] LustreError: 305276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798612 with bad export cookie 193392606143749301 [ 7761.174967] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7761.189230] LustreError: 305276:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7761.542027] Lustre: server umount lustre-MDT0001 complete [ 7768.814768] Lustre: DEBUG MARKER: == conf-sanity test 56b: test target_obd correctness with nonconsecutive MDTs ========================================================== 12:30:19 (1788798619) [ 7770.076342] Lustre: DEBUG MARKER: SKIP: conf-sanity test_56b needs >= 3 MDTs [ 7771.544601] Lustre: DEBUG MARKER: == conf-sanity test 57a: initial registration from failnode should fail (should return errs) ========================================================== 12:30:22 (1788798622) [ 7774.216668] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 7798.295385] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7798.648638] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7798.685195] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7798.864535] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7803.643041] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7811.998039] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7812.092670] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7812.114748] Lustre: 311112:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7816.713770] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7838.319811] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7838.519562] Lustre: Denying initial registration attempt from nid 192.168.206.121@tcp, specified as failover [ 7838.528135] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -99. Please see messages on the MGS. [ 7838.538583] LustreError: 312563:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -99 [ 7838.548608] LustreError: 312563:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 7838.555184] LustreError: 312563:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 7838.628575] Lustre: server umount lustre-OST0000 complete [ 7838.631029] LustreError: 312563:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -99 [ 7845.351189] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 7845.360552] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 7845.376508] Lustre: Skipped 1 previous similar message [ 7845.393018] Lustre: Skipped 3 previous similar messages [ 7851.388311] Lustre: server umount lustre-MDT0000 complete [ 7855.420805] LustreError: 311105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798706 with bad export cookie 193392606143750176 [ 7855.424918] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 7855.437623] LustreError: 311105:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 7855.764754] Lustre: server umount lustre-MDT0001 complete [ 7867.529981] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 7870.295233] Key type lgssc unregistered [ 7870.600933] LNet: 313687:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 7870.610064] LNetError: 313687:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 7870.625877] LNet: Removed LNI 192.168.206.121@tcp [ 7871.361566] Key type .llcrypt unregistered [ 7871.364187] Key type ._llcrypt unregistered [ 7882.732962] Key type ._llcrypt registered [ 7882.734831] Key type .llcrypt registered [ 7882.829869] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 7883.975807] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 7884.052205] alg: No test for adler32 (adler32-zlib) [ 7885.162519] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 7885.438504] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 7887.119235] Key type lgssc registered [ 7888.170725] Lustre: Echo OBD driver; http://www.lustre.org/ [ 7934.683560] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 7943.100760] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 7943.125227] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7944.349388] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 7944.364976] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 7944.504984] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 7948.962702] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7957.185941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 7957.307370] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 7957.336364] Lustre: 316454:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 7957.540235] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 7962.616968] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 7970.396955] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 7975.848656] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 7995.629663] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 7995.852198] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 7996.112870] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8002.262829] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8010.004366] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8015.336484] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8015.348523] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8015.372391] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8015.381883] Lustre: Skipped 1 previous similar message [ 8018.541807] Lustre: server umount lustre-OST0000 complete [ 8024.032201] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8024.040135] LustreError: Skipped 1 previous similar message [ 8024.042532] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8024.068163] Lustre: Skipped 1 previous similar message [ 8024.071873] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8025.568900] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8025.571743] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8030.689269] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8030.699862] Lustre: Skipped 2 previous similar messages [ 8035.810939] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8035.817910] Lustre: Skipped 1 previous similar message [ 8037.344866] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8037.582285] Lustre: server umount lustre-MDT0000 complete [ 8041.760915] LustreError: 316445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788798892 with bad export cookie 8164138105041201470 [ 8041.768295] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8041.781582] LustreError: 316445:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 8042.152894] Lustre: server umount lustre-MDT0001 complete [ 8052.526156] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 8055.064403] Key type lgssc unregistered [ 8055.381378] LNet: 320031:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8055.391450] LNetError: 320031:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8055.406867] LNet: Removed LNI 192.168.206.121@tcp [ 8056.241131] Key type .llcrypt unregistered [ 8056.244553] Key type ._llcrypt unregistered [ 8082.535371] Key type ._llcrypt registered [ 8082.537148] Key type .llcrypt registered [ 8082.677933] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 8084.332034] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8084.450128] alg: No test for adler32 (adler32-zlib) [ 8085.689924] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 8085.943759] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 8087.615184] Key type lgssc registered [ 8089.036107] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8097.306754] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8097.354761] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8099.057554] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8104.189217] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8109.537988] LustreError: 321448:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8109.552035] LustreError: 321448:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8113.147963] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8113.621507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8117.889475] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8125.463895] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8130.761644] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8139.242169] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8139.551917] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8145.572354] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8153.544629] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8168.420835] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8168.431432] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8168.431863] Lustre: Skipped 1 previous similar message [ 8168.454635] Lustre: Skipped 2 previous similar messages [ 8170.907939] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.21@tcp (stopping) [ 8173.537963] LustreError: 321447:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8173.555031] LustreError: 321447:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8173.595647] Lustre: server umount lustre-MDT0000 complete [ 8175.980311] LustreError: 321443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8177.568306] LustreError: 321428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799028 with bad export cookie 14661358831031875069 [ 8177.609147] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8178.656121] LustreError: 322698:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8178.657853] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8178.689857] Lustre: Skipped 1 previous similar message [ 8178.698831] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [ 8181.138795] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.21@tcp (stopping) [ 8182.688888] LustreError: 321443:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8182.716968] LustreError: 321443:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8184.294234] Lustre: server umount lustre-MDT0001 complete [ 8192.642891] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: (null) [ 8203.216634] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8222.696443] Lustre: Evicted from MGS (at 0@lo) after server handle changed from 0x0 to 0xcb779c2d7831149d [ 8222.709160] Lustre: MGC192.168.206.121@tcp: Connection restored to 0@lo (at 0@lo) [ 8223.009939] LustreError: 323282:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8223.114109] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8223.178698] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:193) [ 8227.272446] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8237.263485] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8237.604396] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8238.636090] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8242.486282] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8251.501384] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8253.051729] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8258.715356] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8260.392077] Lustre: DEBUG MARKER: mdc.lustre-MDT0001-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8268.258428] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8268.274768] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8268.295365] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8268.309947] Lustre: Skipped 1 previous similar message [ 8272.193978] Lustre: server umount lustre-OST0000 complete [ 8278.506053] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8278.515116] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8278.536781] Lustre: Skipped 1 previous similar message [ 8278.542911] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8278.545463] Lustre: Skipped 2 previous similar messages [ 8290.271493] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8290.541818] Lustre: server umount lustre-MDT0000 complete [ 8294.711199] LustreError: 321428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799145 with bad export cookie 14661358831031882909 [ 8294.721468] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8294.735387] LustreError: 321428:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 5 previous similar messages [ 8295.177823] Lustre: server umount lustre-MDT0001 complete [ 8306.525934] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 8309.079733] Key type lgssc unregistered [ 8309.293746] LNet: 327747:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8309.304092] LNetError: 327747:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8309.318582] LNet: Removed LNI 192.168.206.121@tcp [ 8310.098746] Key type .llcrypt unregistered [ 8310.101569] Key type ._llcrypt unregistered [ 8334.468243] Key type ._llcrypt registered [ 8334.470353] Key type .llcrypt registered [ 8334.571568] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 8335.394361] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8335.479391] alg: No test for adler32 (adler32-zlib) [ 8336.506497] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 8336.701052] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 8338.352263] Key type lgssc registered [ 8339.483279] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8346.289891] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8346.307935] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8347.561035] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8350.858193] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8357.857387] LustreError: 329162:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8357.880146] LustreError: 329162:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 8359.084879] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8359.531387] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8364.860788] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8374.420336] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8380.782529] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8384.993491] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8385.009362] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8385.035194] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8386.084134] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8386.093356] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8390.587991] Lustre: server umount lustre-MDT0000 complete [ 8391.138043] LustreError: 329158:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8391.160591] LustreError: 329158:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8395.000489] LustreError: 329142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799246 with bad export cookie 9856980855346863817 [ 8395.006944] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8395.017230] LustreError: 329142:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8395.490941] Lustre: server umount lustre-MDT0001 complete [ 8413.013356] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8413.361717] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 8413.406610] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8413.674639] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8418.844429] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8428.802571] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8428.928148] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 8428.993768] Lustre: 331753:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8429.320297] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8434.498363] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8442.681325] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8449.379969] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8459.441409] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8459.753734] LustreError: lustre-OST0000 claims to have registered, but this MGS does not know about it, preventing registration. [ 8459.767567] LustreError: lustre-OST0000: the MGS refuses to allow this server to start: rc = -2. Please see messages on the MGS. [ 8459.779614] LustreError: 333636:0:(tgt_mount.c:2614:server_fill_super()) Unable to start targets: -2 [ 8459.786475] LustreError: 333636:0:(tgt_mount.c:2132:server_put_super()) no obd lustre-OST0000 [ 8459.793325] LustreError: 333636:0:(tgt_mount.c:116:server_deregister_mount()) lustre-OST0000 not registered [ 8459.901550] Lustre: server umount lustre-OST0000 complete [ 8459.909328] LustreError: 333636:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -2 [ 8471.143852] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8471.465522] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [ 8471.696910] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8478.139756] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8480.418154] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:166 to 0x280000401:225) [ 8485.211176] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8493.380607] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8493.478321] Lustre: MGS: Regenerating lustre-OST0001 log by user request: rc = 0 [ 8493.547305] Lustre: lustre-OST0001: new disk, initializing [ 8493.552244] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 8493.557569] Lustre: 335178:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8493.666097] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 8501.426248] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 8501.441648] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 8501.595275] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8501.615275] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 8509.730542] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8517.092863] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8517.111197] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8517.134252] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8517.137732] Lustre: Skipped 1 previous similar message [ 8522.218208] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8522.230153] Lustre: Skipped 1 previous similar message [ 8527.327151] Lustre: lustre-OST0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 8527.347406] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 8527.356023] Lustre: Skipped 1 previous similar message [ 8527.455977] Lustre: server umount lustre-OST0001 complete [ 8531.941278] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 8531.949988] LustreError: Skipped 1 previous similar message [ 8531.956263] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8531.985463] Lustre: Skipped 1 previous similar message [ 8532.448900] LustreError: 335984:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8532.466461] LustreError: 335984:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8537.568392] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8537.569637] LustreError: 334105:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8537.574708] Lustre: Skipped 4 previous similar messages [ 8537.601126] LustreError: 334105:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8537.981676] Lustre: server umount lustre-OST0000 complete [ 8542.176464] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8542.186782] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8542.206501] Lustre: Skipped 1 previous similar message [ 8548.045107] Lustre: server umount lustre-MDT0000 complete [ 8552.127863] LustreError: 332493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799403 with bad export cookie 9856980855346864370 [ 8552.130606] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8552.138968] LustreError: 332493:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 8552.469229] Lustre: server umount lustre-MDT0001 complete [ 8565.917985] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 8569.677461] Key type lgssc unregistered [ 8569.999691] LNet: 337141:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 8570.008137] LNetError: 337141:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 8570.034168] LNet: Removed LNI 192.168.206.121@tcp [ 8571.075186] Key type .llcrypt unregistered [ 8571.077988] Key type ._llcrypt unregistered [ 8648.768950] Key type ._llcrypt registered [ 8648.775240] Key type .llcrypt registered [ 8648.900683] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 8666.032587] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 8667.513078] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 8667.555381] alg: No test for adler32 (adler32-zlib) [ 8668.660692] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 8668.876946] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 8670.559312] Key type lgssc registered [ 8671.680248] Lustre: Echo OBD driver; http://www.lustre.org/ [ 8709.550363] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 8709.582730] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8710.911548] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 8710.950282] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 8711.020658] Lustre: lustre-MDT0000: new disk, initializing [ 8711.110305] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8711.124688] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 8715.771471] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8727.012098] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8727.148385] Lustre: 342078:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 8727.190199] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 8727.197742] Lustre: Skipped 1 previous similar message [ 8727.291686] Lustre: lustre-MDT0001: new disk, initializing [ 8727.429706] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8727.456595] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 8727.468703] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 8731.918372] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8741.342686] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8747.224491] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8754.689058] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8754.962349] Lustre: lustre-OST0000: new disk, initializing [ 8754.974281] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 8754.986893] Lustre: 344050:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 8755.069145] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8761.497737] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8762.964266] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 8762.970232] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 8763.040560] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 8770.690752] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8775.160140] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 8775.351429] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 8780.307400] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 8780.531600] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 8783.850074] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 8783.855101] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8783.859770] LustreError: Skipped 1 previous similar message [ 8783.860863] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 8783.878660] Lustre: Skipped 1 previous similar message [ 8783.900218] Lustre: Skipped 1 previous similar message [ 8788.555892] Lustre: server umount lustre-OST0000 complete [ 8794.099413] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8794.103916] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8794.107238] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 8794.123496] Lustre: Skipped 1 previous similar message [ 8798.685845] Lustre: server umount lustre-MDT0000 complete [ 8802.469453] LustreError: 342073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799653 with bad export cookie 8674342291879001339 [ 8802.475793] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8802.484326] LustreError: 342073:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 8802.978702] Lustre: server umount lustre-MDT0001 complete [ 8812.024682] Lustre: DEBUG MARKER: == conf-sanity test 60b: check mkfs.lustre MDT default features ========================================================== 12:47:42 (1788799662) [ 8820.645577] Lustre: DEBUG MARKER: == conf-sanity test 61a: large xattr ===================== 12:47:51 (1788799671) [ 8829.458820] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8829.947773] LustreError: 346807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8829.963987] LustreError: 346807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8830.038810] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8834.945665] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8835.043931] LustreError: 346808:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8835.066876] LustreError: 346808:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 8840.171994] LustreError: 346807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8844.609140] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8844.916935] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 8850.633939] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8859.958609] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8867.232444] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 8874.864793] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 8875.499916] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 8883.155504] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8891.787685] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 8909.886272] Lustre: DEBUG MARKER: save large xattr of 65536 bytes on trusted.big on /mnt/lustre/f61a.conf-sanity [ 8911.723935] Lustre: DEBUG MARKER: shrink value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8913.402679] Lustre: DEBUG MARKER: grow value of trusted.big on /mnt/lustre/f61a.conf-sanity [ 8915.087350] Lustre: DEBUG MARKER: check value of trusted.big on /mnt/lustre/f61a.conf-sanity after remounting MDS [ 8917.051692] Lustre: Failing over lustre-MDT0000 [ 8917.397596] Lustre: server umount lustre-MDT0000 complete [ 8918.936050] LustreError: 346803:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8920.546188] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8920.568749] Lustre: Skipped 2 previous similar messages [ 8924.011664] LustreError: 346802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8924.031231] LustreError: 346802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 8934.259678] LustreError: 346802:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8934.278857] LustreError: 346802:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 7 previous similar messages [ 8935.482888] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8935.603603] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8935.816115] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8935.862152] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8939.385203] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8940.110957] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8941.041510] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [ 8941.077416] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 8941.108330] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [ 8950.043507] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8951.757043] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 8955.669803] Lustre: DEBUG MARKER: remove large xattr trusted.big from /mnt/lustre/f61a.conf-sanity [ 8957.382822] Lustre: Failing over lustre-MDT0000 [ 8957.702693] Lustre: server umount lustre-MDT0000 complete [ 8959.885562] LustreError: 346804:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [ 8961.505094] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 8961.524536] Lustre: Skipped 1 previous similar message [ 8961.529085] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 8976.944406] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 8977.110923] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 8977.373858] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 8977.432816] Lustre: lustre-MDT0000: in recovery but waiting for the first client to connect [ 8980.346988] Lustre: lustre-MDT0000: Will be in recovery for at least 1:00, or until 2 clients reconnect [ 8981.833207] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 8982.509660] Lustre: lustre-MDT0000-lwp-MDT0001: Connection restored to 0@lo (at 0@lo) [ 8982.515699] Lustre: Skipped 2 previous similar messages [ 8982.537613] Lustre: lustre-MDT0000: Recovery over after 0:02, of 2 clients 2 recovered and 0 were evicted. [ 8982.579494] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [ 8991.616469] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 8993.429296] Lustre: DEBUG MARKER: mdc.lustre-MDT0000-mdc-*.mds_server_uuid in FULL state after 0 sec [ 9002.979630] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9002.981623] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9002.986247] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9002.994934] Lustre: Skipped 3 previous similar messages [ 9003.030378] Lustre: Skipped 3 previous similar messages [ 9006.620882] Lustre: server umount lustre-MDT0000 complete [ 9008.099362] LustreError: 346807:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9008.119799] LustreError: 346807:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 17 previous similar messages [ 9009.903247] LustreError: 346788:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799860 with bad export cookie 8674342291879004048 [ 9009.913614] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9010.311857] Lustre: server umount lustre-MDT0001 complete [ 9024.625579] Lustre: server umount lustre-OST0000 complete [ 9039.138342] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9044.142763] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9052.974838] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9053.275356] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9053.283360] Lustre: Skipped 1 previous similar message [ 9057.833295] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9064.898772] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9070.260692] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9076.668586] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9083.899392] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9084.089137] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [ 9091.869711] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9098.712566] Lustre: server umount lustre-OST0000 complete [ 9104.355189] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9104.362236] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9104.374252] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9109.985306] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9110.002509] Lustre: Skipped 2 previous similar messages [ 9112.231947] Lustre: server umount lustre-MDT0000 complete [ 9115.103316] Lustre: 340304:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788799950/real 1788799950] req@000000008d139a12 x1875692244574208/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788799966 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [ 9115.116846] LustreError: 353173:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9115.161666] LustreError: 353173:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9117.288465] LustreError: 353154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788799968 with bad export cookie 8674342291879004790 [ 9117.307352] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9117.310925] LustreError: 353154:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 6 previous similar messages [ 9118.018333] Lustre: server umount lustre-MDT0001 complete [ 9130.253766] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 9132.872631] Key type lgssc unregistered [ 9133.231447] LNet: 356706:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9133.246913] LNetError: 356706:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9133.268248] LNet: Removed LNI 192.168.206.121@tcp [ 9134.477743] Key type .llcrypt unregistered [ 9134.479592] Key type ._llcrypt unregistered [ 9159.106800] Key type ._llcrypt registered [ 9159.109446] Key type .llcrypt registered [ 9159.233103] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 9160.304653] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9160.465223] alg: No test for adler32 (adler32-zlib) [ 9161.435895] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 9161.658963] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 9163.303175] Key type lgssc registered [ 9164.333951] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9172.088194] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9172.116282] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9173.667957] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9178.334646] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9183.212938] LustreError: 358224:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9183.233299] LustreError: 358224:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9188.322983] LustreError: 358225:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9189.217046] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9189.645120] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9193.881911] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9203.013379] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9209.485549] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9217.492788] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9217.911840] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9218.946904] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [ 9224.756971] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9232.850545] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9244.653967] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9244.667891] Lustre: Skipped 1 previous similar message [ 9244.670038] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9244.685139] Lustre: Skipped 2 previous similar messages [ 9246.761307] Lustre: server umount lustre-MDT0000 complete [ 9249.766844] LustreError: 358219:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9249.799927] LustreError: 358219:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [ 9250.293400] LustreError: 358204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800101 with bad export cookie 10713104947823680419 [ 9250.301098] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9250.312227] LustreError: 358204:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9250.679891] Lustre: server umount lustre-MDT0001 complete [ 9264.968616] Lustre: server umount lustre-OST0000 complete [ 9279.453521] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9279.975561] LustreError: 361939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9280.097294] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9284.931861] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9285.090116] LustreError: 361939:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9285.113747] LustreError: 361939:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9294.158941] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9294.582223] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9299.236138] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9307.312709] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9313.120296] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9319.837820] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9320.258767] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9325.377807] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:193) [ 9326.398972] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9334.127678] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9350.115705] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9350.125626] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9350.131494] Lustre: Skipped 2 previous similar messages [ 9350.145281] Lustre: Skipped 2 previous similar messages [ 9352.831118] Lustre: server umount lustre-MDT0000 complete [ 9355.234858] LustreError: 361934:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9355.250456] LustreError: 361934:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9356.411690] LustreError: 361920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800207 with bad export cookie 10713104947823681539 [ 9356.423234] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9356.427338] LustreError: 361920:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [ 9356.851935] Lustre: server umount lustre-MDT0001 complete [ 9371.150399] Lustre: server umount lustre-OST0000 complete [ 9386.092158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9386.516631] LustreError: 365851:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9386.564805] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9391.405762] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9400.557577] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9405.613363] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9413.407278] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9418.648398] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9424.016290] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9424.284708] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9424.288422] Lustre: Skipped 1 previous similar message [ 9430.017805] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9431.473287] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:163 to 0x280000401:225) [ 9436.459655] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9441.761077] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [ 9441.765779] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9441.777449] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9441.797907] Lustre: Skipped 1 previous similar message [ 9446.756479] Lustre: server umount lustre-OST0000 complete [ 9452.007649] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9452.020245] Lustre: Skipped 1 previous similar message [ 9452.022155] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9452.034057] Lustre: Skipped 2 previous similar messages [ 9457.027884] Lustre: server umount lustre-MDT0000 complete [ 9457.121092] LustreError: 367814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9457.137871] LustreError: 367814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9460.679950] LustreError: 365832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800311 with bad export cookie 10713104947823682561 [ 9460.682495] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9460.694088] LustreError: 365832:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [ 9461.012686] Lustre: server umount lustre-MDT0001 complete [ 9470.252149] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 9472.665721] Key type lgssc unregistered [ 9472.964517] LNet: 369387:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9472.976083] LNetError: 369387:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9472.989785] LNet: Removed LNI 192.168.206.121@tcp [ 9473.635532] Key type .llcrypt unregistered [ 9473.637622] Key type ._llcrypt unregistered [ 9496.880955] Key type ._llcrypt registered [ 9496.884472] Key type .llcrypt registered [ 9496.984713] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 9498.041784] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9498.100094] alg: No test for adler32 (adler32-zlib) [ 9499.150777] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 9499.392242] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 9501.079365] Key type lgssc registered [ 9502.182564] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9509.099266] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9509.112445] LDISKFS-fs (dm-0): mounted filesystem without journal. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9509.120947] LustreError: 370827:0:(osd_handler.c:8547:osd_mount()) lustre-MDT0000-osd: device /dev/mapper/mds1_flakey is mounted w/o journal [ 9509.133430] LustreError: 370827:0:(obd_config.c:845:class_setup()) setup lustre-MDT0000-osd failed (-22) [ 9509.138315] LustreError: 370827:0:(obd_mount.c:259:lustre_start_simple()) lustre-MDT0000-osd setup error -22 [ 9509.142870] LustreError: 370827:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/mds1_flakey: -22 [ 9509.147433] LustreError: 370827:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9515.385626] LDISKFS-fs (dm-2): mounted filesystem without journal. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9515.395522] LustreError: 371225:0:(osd_handler.c:8547:osd_mount()) lustre-OST0000-osd: device /dev/mapper/ost1_flakey is mounted w/o journal [ 9515.417481] LustreError: 371225:0:(obd_config.c:845:class_setup()) setup lustre-OST0000-osd failed (-22) [ 9515.423194] LustreError: 371225:0:(obd_mount.c:259:lustre_start_simple()) lustre-OST0000-osd setup error -22 [ 9515.428284] LustreError: 371225:0:(tgt_mount.c:2582:server_fill_super()) Unable to start osd on /dev/mapper/ost1_flakey: -22 [ 9515.436880] LustreError: 371225:0:(super25.c:179:lustre_fill_super()) llite: Unable to mount : rc = -22 [ 9531.758892] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 9534.264195] Key type lgssc unregistered [ 9534.570896] LNet: 372232:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9534.578548] LNetError: 372232:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9534.607078] LNet: Removed LNI 192.168.206.121@tcp [ 9535.429281] Key type .llcrypt unregistered [ 9535.433094] Key type ._llcrypt unregistered [ 9552.663697] Key type ._llcrypt registered [ 9552.665344] Key type .llcrypt registered [ 9552.880518] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [ 9565.840544] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 9566.691666] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9566.833895] alg: No test for adler32 (adler32-zlib) [ 9567.851846] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 9568.055109] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 9569.735168] Key type lgssc registered [ 9570.833392] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9604.435040] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9604.449158] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9605.664803] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9605.701742] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [ 9605.813850] Lustre: lustre-MDT0000: new disk, initializing [ 9605.898240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9605.928064] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [ 9610.139668] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9619.792784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9619.996960] Lustre: 375369:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9620.117288] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [ 9620.124338] Lustre: Skipped 1 previous similar message [ 9620.198704] Lustre: lustre-MDT0001: new disk, initializing [ 9620.288365] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9620.333147] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [ 9620.344359] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [ 9624.769678] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9633.412525] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9638.292825] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9645.755502] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9646.120255] Lustre: lustre-OST0000: new disk, initializing [ 9646.123312] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [ 9646.131713] Lustre: 377352:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9646.265399] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9652.847538] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [ 9652.865337] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [ 9652.983799] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [ 9653.360979] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9663.797996] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9669.016985] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [ 9669.342381] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [ 9673.986287] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [ 9674.191147] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [ 9678.823407] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9678.838340] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9678.855290] Lustre: Skipped 1 previous similar message [ 9678.866417] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9678.870362] Lustre: Skipped 1 previous similar message [ 9682.476847] Lustre: server umount lustre-OST0000 complete [ 9687.011089] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [ 9687.022889] LustreError: Skipped 1 previous similar message [ 9687.031038] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9687.060374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9689.056867] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9689.058498] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9689.095291] Lustre: Skipped 1 previous similar message [ 9694.177076] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9694.198225] Lustre: Skipped 1 previous similar message [ 9698.279304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [ 9698.287604] Lustre: Skipped 1 previous similar message [ 9700.319149] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9700.456984] Lustre: server umount lustre-MDT0000 complete [ 9703.690850] LustreError: 375363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800554 with bad export cookie 16600458565834301496 [ 9703.697391] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9703.705150] LustreError: 375363:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9704.083856] Lustre: server umount lustre-MDT0001 complete [ 9712.772800] Lustre: DEBUG MARKER: SKIP: conf-sanity test_63 skipping excluded test 63 [ 9714.997188] Lustre: DEBUG MARKER: == conf-sanity test 64: check lfs df --lazy ============== 13:02:45 (1788800565) [ 9721.838971] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9722.525208] LustreError: 379813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9722.546284] LustreError: 379813:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [ 9722.633264] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9727.969843] LustreError: 379814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9728.036879] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9732.074750] LustreError: 379813:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9737.185364] LustreError: 379814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9738.439585] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9738.689093] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9742.620619] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9750.057766] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9755.740132] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [ 9763.488367] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9763.805809] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [ 9771.004625] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9779.245866] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9785.181102] LDISKFS-fs (dm-3): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [ 9785.410394] Lustre: lustre-OST0001: new disk, initializing [ 9785.416179] Lustre: srv-lustre-OST0001: No data found on store. Initialize space. [ 9785.421849] Lustre: 382648:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [ 9785.518358] Lustre: lustre-OST0001: Imperative Recovery not enabled, recovery window 60-180 [ 9792.492110] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9792.585491] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x00000002c0000400-0x0000000300000400]:1:ost [ 9792.590883] Lustre: cli-lustre-OST0001-super: Allocated super-sequence [0x00000002c0000400-0x0000000300000400]:1:ost] [ 9792.680662] Lustre: lustre-OST0001-osc-MDT0000: update sequence from 0x100010000 to 0x2c0000401 [ 9802.082277] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0001-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [ 9808.354623] LustreError: lustre-OST0001-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [ 9808.360601] Lustre: lustre-OST0001-osc-MDT0001: Connection to lustre-OST0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9808.369690] Lustre: lustre-OST0001: Not available for connect from 0@lo (stopping) [ 9808.382729] Lustre: Skipped 1 previous similar message [ 9808.392893] Lustre: Skipped 1 previous similar message [ 9812.334190] Lustre: server umount lustre-OST0001 complete [ 9813.476198] LustreError: 381635:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9813.500503] LustreError: 381635:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [ 9823.724544] LustreError: 381636:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-OST0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9823.750551] LustreError: 381636:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [ 9828.835150] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9828.856993] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [ 9828.865356] Lustre: Skipped 2 previous similar messages [ 9843.167262] Lustre: lustre-OST0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9843.444848] Lustre: server umount lustre-OST0000 complete [ 9849.322264] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [ 9849.342107] Lustre: Skipped 2 previous similar messages [ 9854.090682] Lustre: server umount lustre-MDT0000 complete [ 9854.438947] LustreError: 379814:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [ 9854.459561] LustreError: 379814:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 6 previous similar messages [ 9857.814046] LustreError: 379794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800708 with bad export cookie 16600458565834302378 [ 9857.824023] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [ 9857.839328] LustreError: 379794:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [ 9872.352554] Lustre: lustre-MDT0001 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [ 9873.106329] Lustre: server umount lustre-MDT0001 complete [ 9887.616707] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [ 9890.801440] Key type lgssc unregistered [ 9891.175955] LNet: 384641:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [ 9891.186671] LNetError: 384641:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [ 9892.265920] LNet: Removed LNI 192.168.206.121@tcp [ 9893.100039] Key type .llcrypt unregistered [ 9893.102633] Key type ._llcrypt unregistered [ 9945.321974] Key type ._llcrypt registered [ 9945.325414] Key type .llcrypt registered [ 9945.445930] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [ 9946.765192] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [ 9946.806225] alg: No test for adler32 (adler32-zlib) [ 9947.793090] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [ 9947.978808] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [ 9949.631701] Key type lgssc registered [ 9950.815718] Lustre: Echo OBD driver; http://www.lustre.org/ [ 9959.815081] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [ 9959.843608] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9961.064957] Lustre: MGS: Logs for fs lustre were removed by user request. All servers must be restarted in order to regenerate the logs: rc = 0 [ 9961.090480] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [ 9961.289880] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [ 9966.430522] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9976.038029] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [ 9976.138939] Lustre: MGS: Regenerating lustre-MDT0001 log by user request: rc = 0 [ 9976.177247] Lustre: 387441:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [ 9976.426947] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [ 9981.151694] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [ 9989.471042] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [ 9996.676247] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10001.888718] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10001.902493] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10001.930734] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10003.428243] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10003.430175] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10005.661950] Lustre: server umount lustre-MDT0000 complete [10010.761192] LustreError: 387432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800861 with bad export cookie 6168629237514140528 [10010.766697] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10010.779575] LustreError: 387432:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10011.273410] Lustre: server umount lustre-MDT0001 complete [10021.465957] Lustre: DEBUG MARKER: == conf-sanity test 66: replace nids ===================== 13:07:52 (1788800872) [10028.292942] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10028.753553] LustreError: 389952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10028.767647] LustreError: 389952:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10028.815232] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10032.883552] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10034.147214] LustreError: 389953:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10039.271633] LustreError: 389952:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10040.875077] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10041.166796] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10046.274384] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10055.234080] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10061.782578] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10068.917839] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10069.233356] Lustre: MGS: Regenerating lustre-OST0000 log by user request: rc = 0 [10069.475747] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10077.108611] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10084.777576] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10091.733918] Lustre: Permanently deactivating lustre-OST0000 [10091.747679] Lustre: Setting parameter lustre-OST0000-osc.osc.active=0 in log lustre-client [10101.287277] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10102.728552] LustreError: 392983:0:(mgs_llog.c:2043:only_mgs_is_running()) MGS: node 0@lo still connected during replace_nids connect_flags:a000011001002060 [10102.737329] LustreError: 392983:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10102.744391] LustreError: 392983:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10105.836220] LustreError: 393032:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10105.846842] LustreError: 393032:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10113.829271] Lustre: server umount lustre-OST0000 complete [10116.187570] LustreError: 393283:0:(mgs_llog.c:2098:mgs_replace_nids()) Only MGS is allowed to be started [10116.197650] LustreError: 393283:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-OST0000': rc = -115 [10118.120659] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10118.132294] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10118.156395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10118.161125] Lustre: Skipped 1 previous similar message [10122.208310] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10122.208697] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10122.223616] Lustre: Skipped 1 previous similar message [10127.329395] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10127.339089] Lustre: Skipped 1 previous similar message [10127.399238] Lustre: server umount lustre-MDT0000 complete [10130.754161] LustreError: 389933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788800981 with bad export cookie 6168629237514141109 [10130.761414] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10130.762151] LustreError: 389933:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10140.317884] Lustre: server umount lustre-MDT0001 complete [10149.471951] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10153.802311] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10156.548996] LustreError: 394443:0:(mgs_handler.c:1248:mgs_iocontrol()) MGS: error replacing NIDs for 'lustre-WRONG0000': rc = -6 [10162.681179] Lustre: 394782:0:(mgs_llog.c:1808:mgs_replace_nids_handler()) Previous failover is deleted, but new one is not set. This means you configure system without failover or passed wrong replace_nids command parameters. Device lustre-MDT0000, passed nids 192.168.206.121@tcp [10164.756809] Lustre: server umount MGS complete [10178.215184] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10178.949673] LustreError: 395507:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10179.006393] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10179.016915] Lustre: Skipped 1 previous similar message [10179.124778] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10184.162243] LustreError: 395508:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10184.177385] LustreError: 395508:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10184.344783] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10192.935482] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10193.302021] Lustre: setting import lustre-OST0000_UUID INACTIVE by administrator request [10198.618348] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10206.414859] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10212.559806] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10214.507750] Lustre: Permanently reactivating lustre-OST0000 [10214.531909] Lustre: Modifying parameter lustre-OST0000-osc.osc.active=1 in log lustre-client [10214.541598] Lustre: Skipped 2 previous similar messages [10241.503232] LustreError: 397269:0:(osp_dev.c:699:osp_process_config()) lustre-OST0000-osc-MDT0001: unknown param osc.active=1 [10249.428660] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10249.786491] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10249.794473] Lustre: Skipped 1 previous similar message [10255.767632] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10263.695503] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10270.732510] Lustre: server umount lustre-OST0000 complete [10275.307191] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10275.313323] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10275.324253] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10288.613957] Lustre: 386765:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788801123/real 1788801123] req@00000000273fbb1f x1875693586710912/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788801139 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10288.648113] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10288.659989] Lustre: Skipped 1 previous similar message [10289.120365] Lustre: lustre-MDT0000 is waiting for obd_unlinked_exports more than 8 seconds. The obd refcount = 3. Is it stuck? [10289.294589] Lustre: server umount lustre-MDT0000 complete [10292.816966] LustreError: 395488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801143 with bad export cookie 6168629237514142159 [10292.821834] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10292.837504] LustreError: 395488:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10293.169601] Lustre: server umount lustre-MDT0001 complete [10303.447509] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [10306.153902] Key type lgssc unregistered [10306.442427] LNet: 399877:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10306.461618] LNetError: 399877:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10307.504894] LNet: Removed LNI 192.168.206.121@tcp [10308.205421] Key type .llcrypt unregistered [10308.207668] Key type ._llcrypt unregistered [10327.145230] Key type ._llcrypt registered [10327.150058] Key type .llcrypt registered [10327.271279] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [10342.480678] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [10343.767509] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10343.792085] alg: No test for adler32 (adler32-zlib) [10344.926703] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [10345.196408] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [10346.855159] Key type lgssc registered [10347.851090] Lustre: Echo OBD driver; http://www.lustre.org/ [10379.178676] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10379.204137] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10380.399590] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [10380.426727] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [10380.506694] Lustre: lustre-MDT0000: new disk, initializing [10380.591538] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10380.614857] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [10384.568808] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10394.314512] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10394.411704] Lustre: 403033:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [10394.470806] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [10394.474519] Lustre: Skipped 1 previous similar message [10394.552113] Lustre: lustre-MDT0001: new disk, initializing [10394.627658] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10394.667207] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [10394.679624] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [10398.227646] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10405.960955] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10411.003219] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10416.898211] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10417.157297] Lustre: lustre-OST0000: new disk, initializing [10417.161871] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [10417.167240] Lustre: 405000:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [10417.235624] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10422.770378] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10424.898859] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [10424.909885] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [10424.998535] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [10431.141968] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10436.536831] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [10436.707938] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [10440.626347] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [10440.880624] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [10445.798514] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10445.801224] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10445.806755] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10445.809500] LustreError: Skipped 1 previous similar message [10445.846453] Lustre: Skipped 1 previous similar message [10448.676936] Lustre: server umount lustre-OST0000 complete [10456.033020] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10456.033710] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10456.065715] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10456.088193] Lustre: Skipped 2 previous similar messages [10458.547455] Lustre: server umount lustre-MDT0000 complete [10461.560428] LustreError: 403828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801312 with bad export cookie 1570904254227217995 [10461.569302] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10461.579920] LustreError: 403828:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10461.850912] Lustre: server umount lustre-MDT0001 complete [10469.795795] Lustre: DEBUG MARKER: SKIP: conf-sanity test_67 skipping excluded test 67 [10471.308834] Lustre: DEBUG MARKER: == conf-sanity test 68: be able to reserve specific sequences in FLDB ========================================================== 13:15:22 (1788801322) [10477.303681] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10477.607290] LustreError: 407457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10477.623545] LustreError: 407457:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10477.661982] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10481.737382] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10482.657639] LustreError: 407458:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10487.790273] LustreError: 407457:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10490.383246] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10490.783507] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10495.165233] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10501.974435] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10507.600226] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10513.419571] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10513.763802] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10519.174378] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10525.688493] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10527.446348] Lustre: ctl-lustre-MDT0000: [0x00000002c0000400-0x0000000300000400]:0:mdt sequences allocated: rc = 0 [10529.932071] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000300000400-0x0000000340000400]:0:mdt [10536.416468] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10536.427756] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10536.447844] Lustre: Skipped 1 previous similar message [10536.459641] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10540.829476] Lustre: server umount lustre-OST0000 complete [10547.172081] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10547.180076] LustreError: Skipped 1 previous similar message [10547.185789] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10547.205180] Lustre: Skipped 1 previous similar message [10547.209884] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10547.214617] Lustre: Skipped 2 previous similar messages [10552.755204] Lustre: server umount lustre-MDT0000 complete [10552.803136] LustreError: 407454:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10552.832747] LustreError: 407454:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10556.118826] LustreError: 407440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801407 with bad export cookie 1570904254227218877 [10556.121176] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10556.134668] LustreError: 407440:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [10556.346302] Lustre: server umount lustre-MDT0001 complete [10565.525375] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [10567.618063] Key type lgssc unregistered [10567.831649] LNet: 411185:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10567.839154] LNetError: 411185:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10567.852655] LNet: Removed LNI 192.168.206.121@tcp [10568.601744] Key type .llcrypt unregistered [10568.603319] Key type ._llcrypt unregistered [10589.798108] Key type ._llcrypt registered [10589.800462] Key type .llcrypt registered [10589.874612] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [10590.145334] Key type .llcrypt unregistered [10590.150602] Key type ._llcrypt unregistered [10604.021677] Key type ._llcrypt registered [10604.023299] Key type .llcrypt registered [10604.096524] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [10604.754899] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10604.852238] alg: No test for adler32 (adler32-zlib) [10605.914882] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [10606.145991] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [10607.879154] Key type lgssc registered [10609.074937] Lustre: Echo OBD driver; http://www.lustre.org/ [10616.764483] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10616.839042] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10618.218591] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10622.648060] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10628.579354] LustreError: 413380:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10628.600958] LustreError: 413380:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10630.620605] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10630.905288] LustreError: 413381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10630.936493] LustreError: 413381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10630.950174] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10636.262107] LustreError: 413376:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10636.280424] LustreError: 413376:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10636.762036] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10641.380675] LustreError: 413375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10641.394382] LustreError: 413375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10643.168477] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10643.432323] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:65) [10648.553764] LustreError: 413381:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10648.575329] LustreError: 413381:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [10648.936164] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10649.414118] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10653.422439] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10663.422315] Lustre: server umount lustre-OST0000 complete [10669.039990] LustreError: lustre-MDT0000-osp-MDT0001: operation out_update to node 0@lo failed: rc = -107 [10669.056285] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10669.092776] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10670.049445] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10670.053536] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10675.170468] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10675.181612] Lustre: Skipped 2 previous similar messages [10677.347580] Lustre: server umount lustre-MDT0000 complete [10680.287108] Lustre: 412692:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788801515/real 1788801515] req@000000008ba5be9a x1875694276601472/t0(0) o13->lustre-OST0000-osc-MDT0001@0@lo:7/4 lens 224/368 e 0 to 1 dl 1788801531 ref 1 fl Rpc:XQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [10680.315430] LustreError: 413375:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10680.350404] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10680.397735] LustreError: 413375:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10681.402640] LustreError: 413360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801532 with bad export cookie 6288944590710108849 [10681.405050] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10681.423800] LustreError: 413360:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [10681.694291] Lustre: server umount lustre-MDT0001 complete [10693.735127] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [10697.046179] Key type lgssc unregistered [10697.230510] LNet: 416533:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10697.234886] LNetError: 416533:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10697.246269] LNet: Removed LNI 192.168.206.121@tcp [10697.923462] Key type .llcrypt unregistered [10697.926184] Key type ._llcrypt unregistered [10721.725325] Key type ._llcrypt registered [10721.727682] Key type .llcrypt registered [10721.828478] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [10722.771661] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10722.797915] alg: No test for adler32 (adler32-zlib) [10724.081170] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [10724.417446] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [10726.071147] Key type lgssc registered [10727.416945] Lustre: Echo OBD driver; http://www.lustre.org/ [10735.727109] Lustre: lustre-OST0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10735.752734] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10751.327470] LustreError: Failed to get MGS log lustre-sptlrpc and no local copy. [10756.447516] LustreError: 417917:0:(mgc_request.c:233:do_config_log_add()) MGC192.168.206.121@tcp: failed processing log, type 4: rc = -110 [10783.071232] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10789.942180] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10802.116846] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10811.306955] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10811.938276] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10812.054486] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:97) [10815.492931] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10816.995596] LustreError: 417943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10817.023184] LustreError: 417943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10818.020025] LustreError: 418955:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10818.045943] LustreError: 418955:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10823.139649] LustreError: 417943:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10823.159865] LustreError: 417943:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10823.947306] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10824.316772] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10828.683290] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10836.221390] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10842.294621] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10849.762730] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10849.767443] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10849.775387] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10850.277550] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10850.283036] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10850.292876] Lustre: Skipped 1 previous similar message [10851.639449] Lustre: server umount lustre-OST0000 complete [10855.395145] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10855.406706] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10855.422542] Lustre: Skipped 1 previous similar message [10857.309379] Lustre: server umount lustre-MDT0000 complete [10860.515913] LustreError: 418959:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10860.535642] LustreError: 418959:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [10860.610925] LustreError: 417924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801711 with bad export cookie 8707409750312639082 [10860.617639] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10860.621213] LustreError: 417924:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [10860.917498] Lustre: server umount lustre-MDT0001 complete [10871.855136] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [10874.774805] Key type lgssc unregistered [10875.103252] LNet: 421489:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [10875.120229] LNetError: 421489:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [10875.154758] LNet: Removed LNI 192.168.206.121@tcp [10876.112508] Key type .llcrypt unregistered [10876.114783] Key type ._llcrypt unregistered [10898.698498] Key type ._llcrypt registered [10898.700545] Key type .llcrypt registered [10898.783134] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [10899.690966] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [10899.738299] alg: No test for adler32 (adler32-zlib) [10900.859688] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [10901.098174] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [10902.815341] Key type lgssc registered [10903.907304] Lustre: Echo OBD driver; http://www.lustre.org/ [10911.312802] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [10911.334655] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10912.911247] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [10917.597244] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10922.469645] LustreError: 422887:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10922.492044] LustreError: 422887:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [10927.585370] LustreError: 422886:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10927.878781] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [10928.528345] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [10933.760731] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10941.765321] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [10946.878968] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [10953.618045] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [10953.913449] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [10956.985480] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:129) [10960.626786] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [10969.290890] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [10974.688889] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [10974.695635] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10974.708214] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10977.212594] Lustre: lustre-MDT0000: Not available for connect from 192.168.206.21@tcp (stopping) [10978.788877] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10978.788895] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [10980.303191] Lustre: server umount lustre-MDT0000 complete [10982.257062] LustreError: 422882:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 192.168.206.21@tcp (no target). If you are running an HA pair check that the target is mounted on the other server. [10983.649114] LustreError: 422867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 192.168.206.21@tcp arrived at 1788801834 with bad export cookie 17840571065236498107 [10983.687537] LustreError: 422867:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10986.353240] LustreError: 422868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788801837 with bad export cookie 17840571065236497218 [10986.374980] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [10986.389183] LustreError: 422868:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [10989.025330] LustreError: 423615:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10989.025806] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [10989.083035] LustreError: 423615:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [10989.092899] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [10989.097665] Lustre: Skipped 1 previous similar message [10989.100834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [10989.103852] Lustre: Skipped 2 previous similar messages [10992.745509] Lustre: server umount lustre-OST0000 complete [10996.197461] LustreError: 422881:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [10996.234027] LustreError: 422881:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [11001.116513] Lustre: server umount lustre-MDT0001 complete [11011.604371] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [11015.217676] Key type lgssc unregistered [11015.552906] LNet: 426575:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11015.577776] LNetError: 426575:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11015.607554] LNet: Removed LNI 192.168.206.121@tcp [11016.244984] Key type .llcrypt unregistered [11016.247637] Key type ._llcrypt unregistered [11041.626407] Key type ._llcrypt registered [11041.635737] Key type .llcrypt registered [11041.715700] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [11042.612578] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11042.645492] alg: No test for adler32 (adler32-zlib) [11043.811928] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [11044.064347] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [11045.727167] Key type lgssc registered [11046.732661] Lustre: Echo OBD driver; http://www.lustre.org/ [11053.584601] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11053.608731] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11055.030825] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11059.123513] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [11065.314450] LustreError: 427992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11065.329452] LustreError: 427992:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [11067.259415] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11067.581427] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11072.045430] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [11078.736399] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11083.919632] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11089.445263] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11089.753799] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11094.461270] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [11100.036241] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:35 to 0x280000401:161) [11100.323341] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11105.260125] Lustre: lustre-MDT0001-osp-MDT0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11105.273502] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11105.280209] Lustre: Skipped 1 previous similar message [11108.233605] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.21@tcp (stopping) [11110.370452] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11110.378268] Lustre: Skipped 1 previous similar message [11110.908229] Lustre: server umount lustre-MDT0001 complete [11115.501975] LustreError: 427993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11115.521382] LustreError: 427993:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11120.609314] LustreError: 427992:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11120.612608] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11120.643743] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11120.654758] Lustre: Skipped 1 previous similar message [11120.661546] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11122.475271] Lustre: server umount lustre-OST0000 complete [11125.736325] LustreError: 427993:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11126.506952] Lustre: server umount lustre-MDT0000 complete [11139.810762] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [11142.408684] Key type lgssc unregistered [11142.638718] LNet: 431680:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11142.645891] LNetError: 431680:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11143.720968] LNet: Removed LNI 192.168.206.121@tcp [11144.444673] Key type .llcrypt unregistered [11144.448994] Key type ._llcrypt unregistered [11166.700042] Key type ._llcrypt registered [11166.704597] Key type .llcrypt registered [11166.814139] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [11181.183351] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [11182.244184] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11182.261410] alg: No test for adler32 (adler32-zlib) [11183.239747] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [11183.540338] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [11185.191163] Key type lgssc registered [11186.057755] Lustre: Echo OBD driver; http://www.lustre.org/ [11220.112632] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [11235.344208] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11235.363115] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11236.527770] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11236.553948] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11236.633081] Lustre: lustre-MDT0000: new disk, initializing [11236.689175] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11236.701489] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11241.379969] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [11255.739652] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11255.987459] Lustre: lustre-OST0000: new disk, initializing [11255.994527] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11256.000357] Lustre: Skipped 1 previous similar message [11256.004777] Lustre: 436715:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11256.093617] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11261.681858] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [11263.132351] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [11263.138994] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [11263.187093] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [11270.289810] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11284.761345] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11284.904884] Lustre: 435694:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11284.962420] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11285.045692] Lustre: lustre-MDT0001: new disk, initializing [11285.170212] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11285.218473] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:1:mdt [11285.234386] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:1:mdt] [11289.017457] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [11297.266621] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid 50 [11297.441426] Lustre: DEBUG MARKER: osp.lustre-MDT0000-osp-MDT0001.mdt_server_uuid in FULL state after 0 sec [11300.875525] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state (FULL|IDLE) osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid 50 [11301.068634] Lustre: DEBUG MARKER: osp.lustre-MDT0001-osp-MDT0000.mdt_server_uuid in FULL state after 0 sec [11306.896552] Lustre: lustre-MDT0001: Not available for connect from 192.168.206.21@tcp (stopping) [11307.489600] LustreError: lustre-MDT0001-osp-MDT0000: operation mds_statfs to node 0@lo failed: rc = -107 [11307.494937] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11307.512920] Lustre: lustre-MDT0001: Not available for connect from 0@lo (stopping) [11307.516389] Lustre: Skipped 1 previous similar message [11307.538615] Lustre: Skipped 1 previous similar message [11311.394865] Lustre: server umount lustre-MDT0001 complete [11317.734796] LustreError: 435704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11317.774206] LustreError: 435704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 5 previous similar messages [11322.856490] LustreError: 435704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11322.881717] LustreError: 435704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11327.975092] LustreError: 435704:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11327.987722] LustreError: 435704:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11330.019037] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11330.029355] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11330.043575] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11333.090990] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11333.094514] LustreError: 435707:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11334.318992] Lustre: server umount lustre-OST0000 complete [11343.871667] Lustre: server umount lustre-MDT0000 complete [11354.859870] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [11357.257869] Key type lgssc unregistered [11357.495540] LNet: 440462:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [11357.502308] LNetError: 440462:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [11357.522170] LNet: Removed LNI 192.168.206.121@tcp [11358.273295] Key type .llcrypt unregistered [11358.279428] Key type ._llcrypt unregistered [11388.702426] Key type ._llcrypt registered [11388.705983] Key type .llcrypt registered [11388.804066] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [11389.051626] Key type .llcrypt unregistered [11389.054552] Key type ._llcrypt unregistered [11402.215209] Key type ._llcrypt registered [11402.222600] Key type .llcrypt registered [11402.338971] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [11403.167422] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [11403.185776] alg: No test for adler32 (adler32-zlib) [11404.198242] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [11404.396327] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [11406.063149] Key type lgssc registered [11407.000876] Lustre: Echo OBD driver; http://www.lustre.org/ [11428.091720] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [11428.116986] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11429.353584] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [11429.380595] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [11429.445484] Lustre: lustre-MDT0000: new disk, initializing [11429.498816] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11429.512608] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [11432.909172] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [11441.482620] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11441.581669] Lustre: 443818:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [11441.645370] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [11441.648288] Lustre: Skipped 1 previous similar message [11441.718569] Lustre: lustre-MDT0001: new disk, initializing [11441.815270] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11441.874291] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [11441.889272] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [11445.817316] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [11453.619784] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11458.676730] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11464.768518] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11465.110750] Lustre: lustre-OST0000: new disk, initializing [11465.122445] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [11465.133782] Lustre: 445785:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [11465.266647] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11471.012770] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [11472.989901] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [11473.001704] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [11473.106600] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [11480.032254] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11488.227181] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [11488.232188] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11488.241148] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11488.743357] Lustre: lustre-MDT0000-lwp-OST0000: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11488.745337] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11488.755568] Lustre: Skipped 1 previous similar message [11488.767057] Lustre: Skipped 1 previous similar message [11491.918738] Lustre: server umount lustre-MDT0000 complete [11495.136485] LustreError: 443811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802346 with bad export cookie 4329083121689413486 [11495.141749] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11495.151595] LustreError: 443811:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [11495.426825] Lustre: server umount lustre-MDT0001 complete [11509.685067] Lustre: server umount lustre-OST0000 complete [11522.928683] Lustre: DEBUG MARKER: == conf-sanity test 73a: failnode to update from mountdata properly ========================================================== 13:32:53 (1788802373) [11532.068490] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11532.732329] LustreError: 447816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11532.763988] LustreError: 447816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [11532.844916] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [11537.522423] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [11537.890105] LustreError: 447817:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11541.996490] LustreError: 447816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11545.440784] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [11545.844702] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [11549.596301] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [11555.987636] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [11560.836732] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [11579.627760] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [11579.849459] Lustre: Found index 0 for lustre-OST0000, updating log [11580.086296] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [11587.601171] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [11595.710972] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [11603.659441] Lustre: lustre-OST0000: new connection from lustre-MDT0000-mdtlov (cleaning up unused objects from 0x280000401:5 to 0x280000401:33) [11614.178319] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11614.182893] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [11614.198026] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [11614.216455] Lustre: Skipped 1 previous similar message [11618.420854] Lustre: server umount lustre-OST0000 complete [11636.706610] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [11636.735685] Lustre: Skipped 1 previous similar message [11636.749232] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [11636.761605] Lustre: Skipped 3 previous similar messages [11640.343115] Lustre: server umount lustre-MDT0000 complete [11641.824766] LustreError: 447816:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [11641.835517] LustreError: 447816:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [11642.975636] LNet: 1 local NIs in recovery (showing 1): 192.168.206.121@tcp [11644.390795] LustreError: 448552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788802495 with bad export cookie 4329083121689414879 [11644.406646] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [11644.419341] LustreError: 448552:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [11644.742409] Lustre: server umount lustre-MDT0001 complete [11652.463808] Lustre: DEBUG MARKER: == conf-sanity test 73b: Large failnode NID list in mountdata ========================================================== 13:35:03 (1788802503) [12268.995164] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12269.386467] LustreError: 454246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12269.409826] LustreError: 454246:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [12269.466598] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12273.876626] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [12274.660163] LustreError: 454247:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12279.779483] LustreError: 454246:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12283.423277] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12283.837409] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12288.237694] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [12295.439535] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12300.097965] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12318.978613] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12319.244218] Lustre: Found index 0 for lustre-OST0000, updating log [12319.488364] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12321.887217] LNet: 1 local NIs in recovery (showing 1): 192.168.206.121@tcp [12324.455756] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [12331.547775] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12333.639789] Lustre: server umount lustre-OST0000 complete [12338.338800] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12338.341530] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12338.355038] Lustre: Skipped 2 previous similar messages [12339.435374] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12339.441353] Lustre: Skipped 2 previous similar messages [12340.448579] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12340.455549] Lustre: Skipped 1 previous similar message [12342.499488] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12342.506612] Lustre: Skipped 3 previous similar messages [12343.671631] Lustre: server umount lustre-MDT0000 complete [12344.559624] LustreError: 456716:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [12344.569764] LustreError: 456716:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [12347.024549] LustreError: 454228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803198 with bad export cookie 4329083121689415964 [12347.029148] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12347.034381] LustreError: 454228:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [12347.344348] Lustre: server umount lustre-MDT0001 complete [12375.436383] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [12381.283892] LNet: 1 local NIs in recovery (showing 1): 192.168.206.121@tcp [12384.442558] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [12416.710888] Lustre: DEBUG MARKER: == conf-sanity test 73c: Server mount doesn't fail with > 32 nets ========================================================== 13:47:47 (1788803267) [12433.971341] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [12436.903384] Key type lgssc unregistered [12437.281800] LNet: 460859:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12437.292671] LNetError: 460859:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12531.561131] LNet: Removed LNI 192.168.206.121@tcp [12532.320512] Key type .llcrypt unregistered [12532.323058] Key type ._llcrypt unregistered [12547.486519] Key type ._llcrypt registered [12547.491467] Key type .llcrypt registered [12547.582623] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [12548.307657] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12548.319484] alg: No test for adler32 (adler32-zlib) [12549.427464] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [12549.653937] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [12551.304618] Key type lgssc registered [12552.372191] Lustre: Echo OBD driver; http://www.lustre.org/ [12556.437897] LNet: Added LNI 192.168.206.121@tcp100 [8/256/0/180] [12557.251616] LNet: Added LNI 192.168.206.121@tcp101 [8/256/0/180] [12558.195911] LNet: Added LNI 192.168.206.121@tcp102 [8/256/0/180] [12558.988415] LNet: Added LNI 192.168.206.121@tcp103 [8/256/0/180] [12559.837611] LNet: Added LNI 192.168.206.121@tcp104 [8/256/0/180] [12560.778246] LNet: Added LNI 192.168.206.121@tcp105 [8/256/0/180] [12561.596029] LNet: Added LNI 192.168.206.121@tcp106 [8/256/0/180] [12562.524512] LNet: Added LNI 192.168.206.121@tcp107 [8/256/0/180] [12563.542645] LNet: Added LNI 192.168.206.121@tcp108 [8/256/0/180] [12564.401331] LNet: Added LNI 192.168.206.121@tcp109 [8/256/0/180] [12565.321631] LNet: Added LNI 192.168.206.121@tcp110 [8/256/0/180] [12566.155369] LNet: Added LNI 192.168.206.121@tcp111 [8/256/0/180] [12566.939519] LNet: Added LNI 192.168.206.121@tcp112 [8/256/0/180] [12567.821279] LNet: Added LNI 192.168.206.121@tcp113 [8/256/0/180] [12568.796045] LNet: Added LNI 192.168.206.121@tcp114 [8/256/0/180] [12569.787233] LNet: Added LNI 192.168.206.121@tcp115 [8/256/0/180] [12570.644601] LNet: Added LNI 192.168.206.121@tcp116 [8/256/0/180] [12571.574430] LNet: Added LNI 192.168.206.121@tcp117 [8/256/0/180] [12572.456882] LNet: Added LNI 192.168.206.121@tcp118 [8/256/0/180] [12573.586417] LNet: Added LNI 192.168.206.121@tcp119 [8/256/0/180] [12574.484784] LNet: Added LNI 192.168.206.121@tcp120 [8/256/0/180] [12575.492529] LNet: Added LNI 192.168.206.121@tcp121 [8/256/0/180] [12576.446543] LNet: Added LNI 192.168.206.121@tcp122 [8/256/0/180] [12577.269551] LNet: Added LNI 192.168.206.121@tcp123 [8/256/0/180] [12578.218375] LNet: Added LNI 192.168.206.121@tcp124 [8/256/0/180] [12579.099903] LNet: Added LNI 192.168.206.121@tcp125 [8/256/0/180] [12579.966614] LNet: Added LNI 192.168.206.121@tcp126 [8/256/0/180] [12580.781403] LNet: Added LNI 192.168.206.121@tcp127 [8/256/0/180] [12581.509255] LNet: Added LNI 192.168.206.121@tcp128 [8/256/0/180] [12582.267943] LNet: Added LNI 192.168.206.121@tcp129 [8/256/0/180] [12583.226984] LNet: Added LNI 192.168.206.121@tcp130 [8/256/0/180] [12584.170429] LNet: Added LNI 192.168.206.121@tcp131 [8/256/0/180] [12584.909468] LNet: Added LNI 192.168.206.121@tcp132 [8/256/0/180] [12585.563790] LNet: Added LNI 192.168.206.121@tcp133 [8/256/0/180] [12586.331487] LNet: Added LNI 192.168.206.121@tcp134 [8/256/0/180] [12587.234339] LNet: Added LNI 192.168.206.121@tcp135 [8/256/0/180] [12593.485179] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12593.544887] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12594.668915] Lustre: 463889:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.121@tcp: can't add NID 192.168.206.121@tcp130: rc = -75 [12594.913677] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12595.000940] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12595.162948] Lustre: lustre-MDT0000: new disk, initializing [12595.257069] Lustre: 463940:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.121@tcp: can't add NID 192.168.206.121@tcp130: rc = -75 [12595.272207] Lustre: 463940:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [12595.327240] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12595.353363] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12600.372684] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [12611.007215] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12611.178174] Lustre: 463904:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12611.288502] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12611.300275] Lustre: Skipped 1 previous similar message [12611.434535] Lustre: lustre-MDT0001: new disk, initializing [12611.485401] Lustre: 464704:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.121@tcp: can't add NID 192.168.206.121@tcp130: rc = -75 [12611.494400] Lustre: 464704:0:(lustre_peer.c:124:class_add_uuid()) Skipped 5 previous similar messages [12611.705404] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12611.794189] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12611.811362] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12615.926184] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [12625.081947] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12630.304529] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12635.738469] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12636.020847] Lustre: lustre-OST0000: new disk, initializing [12636.025118] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12636.029982] Lustre: 465878:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12636.108940] Lustre: 465883:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.121@tcp: can't add NID 192.168.206.121@tcp130: rc = -75 [12636.116177] Lustre: 465883:0:(lustre_peer.c:124:class_add_uuid()) Skipped 17 previous similar messages [12636.155472] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12640.956858] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [12643.854348] Lustre: 466357:0:(lustre_peer.c:124:class_add_uuid()) 192.168.206.121@tcp: can't add NID 192.168.206.121@tcp130: rc = -75 [12643.864144] Lustre: 466357:0:(lustre_peer.c:124:class_add_uuid()) Skipped 11 previous similar messages [12643.921120] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12643.930206] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12644.011636] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12648.602380] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12654.567826] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12654.569524] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [12654.588305] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12654.600324] LustreError: Skipped 1 previous similar message [12659.684924] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [12659.690874] Lustre: Skipped 1 previous similar message [12660.395214] Lustre: server umount lustre-OST0000 complete [12664.804572] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [12664.818650] Lustre: Skipped 2 previous similar messages [12664.819718] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [12664.838436] Lustre: Skipped 2 previous similar messages [12669.923219] Lustre: server umount lustre-MDT0000 complete [12672.919575] LustreError: 463898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788803523 with bad export cookie 14185982302932345354 [12672.930924] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [12672.933306] LustreError: 463898:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [12673.282658] Lustre: server umount lustre-MDT0001 complete [12683.383862] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [12686.030304] Key type lgssc unregistered [12686.356079] LNet: 467790:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12686.373753] LNetError: 467790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12686.390222] LNet: Removed LNI 192.168.206.121@tcp [12686.392198] LNet: Removed LNI 192.168.206.121@tcp100 [12687.455146] LNet: Removed LNI 192.168.206.121@tcp101 [12687.457118] LNetError: 467790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12687.482161] LNetError: 467790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 2 previous similar messages [12687.496274] LNet: Removed LNI 192.168.206.121@tcp102 [12688.543422] LNet: Removed LNI 192.168.206.121@tcp103 [12688.550552] LNetError: 467790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12688.564207] LNetError: 467790:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Skipped 1 previous similar message [12688.575391] LNet: Removed LNI 192.168.206.121@tcp104 [12688.580325] LNet: Removed LNI 192.168.206.121@tcp105 [12688.585222] LNet: Removed LNI 192.168.206.121@tcp106 [12688.589225] LNet: Removed LNI 192.168.206.121@tcp107 [12688.591689] LNet: Removed LNI 192.168.206.121@tcp108 [12688.595453] LNet: Removed LNI 192.168.206.121@tcp109 [12688.598211] LNet: Removed LNI 192.168.206.121@tcp110 [12688.601582] LNet: Removed LNI 192.168.206.121@tcp111 [12688.605338] LNet: Removed LNI 192.168.206.121@tcp112 [12688.609730] LNet: Removed LNI 192.168.206.121@tcp113 [12688.613911] LNet: Removed LNI 192.168.206.121@tcp114 [12688.623986] LNet: Removed LNI 192.168.206.121@tcp115 [12688.632346] LNet: Removed LNI 192.168.206.121@tcp116 [12688.650544] LNet: Removed LNI 192.168.206.121@tcp117 [12688.655621] LNet: Removed LNI 192.168.206.121@tcp118 [12688.662553] LNet: Removed LNI 192.168.206.121@tcp119 [12688.665397] LNet: Removed LNI 192.168.206.121@tcp120 [12688.668117] LNet: Removed LNI 192.168.206.121@tcp121 [12688.672493] LNet: Removed LNI 192.168.206.121@tcp122 [12688.676528] LNet: Removed LNI 192.168.206.121@tcp123 [12688.680551] LNet: Removed LNI 192.168.206.121@tcp124 [12688.684598] LNet: Removed LNI 192.168.206.121@tcp125 [12688.690525] LNet: Removed LNI 192.168.206.121@tcp126 [12688.695698] LNet: Removed LNI 192.168.206.121@tcp127 [12688.699281] LNet: Removed LNI 192.168.206.121@tcp128 [12688.712353] LNet: Removed LNI 192.168.206.121@tcp129 [12688.735557] LNet: Removed LNI 192.168.206.121@tcp130 [12688.744222] LNet: Removed LNI 192.168.206.121@tcp131 [12688.746169] LNet: Removed LNI 192.168.206.121@tcp132 [12688.755945] LNet: Removed LNI 192.168.206.121@tcp133 [12688.771279] LNet: Removed LNI 192.168.206.121@tcp134 [12688.792224] LNet: Removed LNI 192.168.206.121@tcp135 [12689.731362] Key type .llcrypt unregistered [12689.733258] Key type ._llcrypt unregistered [12718.853275] Key type ._llcrypt registered [12718.860191] Key type .llcrypt registered [12718.984252] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [12734.999948] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [12736.007103] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12736.169583] alg: No test for adler32 (adler32-zlib) [12737.207453] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [12737.425040] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [12739.095188] Key type lgssc registered [12740.225520] Lustre: Echo OBD driver; http://www.lustre.org/ [12771.747490] Lustre: DEBUG MARKER: == conf-sanity test 73e: Mount client with dynamic server NIDs ========================================================== 13:53:42 (1788803622) [12787.337499] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [12789.650625] Key type lgssc unregistered [12789.925590] LNet: 471878:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [12789.930590] LNetError: 471878:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [12789.945600] LNet: Removed LNI 192.168.206.121@tcp [12790.641653] Key type .llcrypt unregistered [12790.643831] Key type ._llcrypt unregistered [12806.388626] Key type ._llcrypt registered [12806.391569] Key type .llcrypt registered [12806.541563] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [12807.653555] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [12807.712077] alg: No test for adler32 (adler32-zlib) [12808.836737] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [12809.132279] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [12810.784584] Key type lgssc registered [12811.982926] Lustre: Echo OBD driver; http://www.lustre.org/ [12821.480200] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [12821.514181] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12822.762652] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [12822.798549] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [12822.875627] Lustre: lustre-MDT0000: new disk, initializing [12823.017383] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [12823.034898] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [12827.237452] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [12838.076940] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [12838.213468] Lustre: 473092:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [12838.288328] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [12838.292312] Lustre: Skipped 1 previous similar message [12838.361680] Lustre: lustre-MDT0001: new disk, initializing [12838.479974] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [12838.528070] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [12838.545697] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [12843.167281] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [12851.968782] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [12857.991885] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [12864.047704] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [12864.397970] Lustre: lustre-OST0000: new disk, initializing [12864.405356] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [12864.417321] Lustre: 475058:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [12864.529377] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [12870.964689] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [12872.737659] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [12872.744863] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [12872.919111] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [12881.643132] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [12885.370837] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing lnet_if_list [12892.411549] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp10 --if ens2 [12892.469967] LNet: Added LNI 192.168.206.121@tcp10 [8/256/0/180] [12900.431672] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp11 --if ens2 [12900.507837] LNet: Added LNI 192.168.206.121@tcp11 [8/256/0/180] [12905.823259] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp12 --if ens2 [12905.848597] LNet: Added LNI 192.168.206.121@tcp12 [8/256/0/180] [12910.451944] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp13 --if ens2 [12910.491387] LNet: Added LNI 192.168.206.121@tcp13 [8/256/0/180] [12915.699283] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp14 --if ens2 [12915.731661] LNet: Added LNI 192.168.206.121@tcp14 [8/256/0/180] [12920.457902] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp15 --if ens2 [12920.491675] LNet: Added LNI 192.168.206.121@tcp15 [8/256/0/180] [12925.621564] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp16 --if ens2 [12925.648697] LNet: Added LNI 192.168.206.121@tcp16 [8/256/0/180] [12929.675111] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp17 --if ens2 [12929.703200] LNet: Added LNI 192.168.206.121@tcp17 [8/256/0/180] [12934.237560] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp18 --if ens2 [12934.256369] LNet: Added LNI 192.168.206.121@tcp18 [8/256/0/180] [12939.363934] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp19 --if ens2 [12939.405770] LNet: Added LNI 192.168.206.121@tcp19 [8/256/0/180] [12944.380367] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp20 --if ens2 [12944.406327] LNet: Added LNI 192.168.206.121@tcp20 [8/256/0/180] [12949.477168] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp21 --if ens2 [12949.522763] LNet: Added LNI 192.168.206.121@tcp21 [8/256/0/180] [12955.478675] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp22 --if ens2 [12955.533840] LNet: Added LNI 192.168.206.121@tcp22 [8/256/0/180] [12961.993157] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp23 --if ens2 [12962.034576] LNet: Added LNI 192.168.206.121@tcp23 [8/256/0/180] [12968.539268] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp24 --if ens2 [12968.589500] LNet: Added LNI 192.168.206.121@tcp24 [8/256/0/180] [12974.812601] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp25 --if ens2 [12974.871605] LNet: Added LNI 192.168.206.121@tcp25 [8/256/0/180] [12981.263477] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp26 --if ens2 [12981.298038] LNet: Added LNI 192.168.206.121@tcp26 [8/256/0/180] [12986.888133] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp27 --if ens2 [12986.915491] LNet: Added LNI 192.168.206.121@tcp27 [8/256/0/180] [12992.803574] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp28 --if ens2 [12992.829108] LNet: Added LNI 192.168.206.121@tcp28 [8/256/0/180] [12999.367716] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp29 --if ens2 [12999.396143] LNet: Added LNI 192.168.206.121@tcp29 [8/256/0/180] [13005.326477] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp30 --if ens2 [13005.355267] LNet: Added LNI 192.168.206.121@tcp30 [8/256/0/180] [13010.895881] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp31 --if ens2 [13010.935736] LNet: Added LNI 192.168.206.121@tcp31 [8/256/0/180] [13016.743556] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp32 --if ens2 [13016.772795] LNet: Added LNI 192.168.206.121@tcp32 [8/256/0/180] [13023.024742] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp33 --if ens2 [13023.067981] LNet: Added LNI 192.168.206.121@tcp33 [8/256/0/180] [13030.473210] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp34 --if ens2 [13030.510884] LNet: Added LNI 192.168.206.121@tcp34 [8/256/0/180] [13039.267323] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp35 --if ens2 [13039.310235] LNet: Added LNI 192.168.206.121@tcp35 [8/256/0/180] [13049.185714] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp36 --if ens2 [13049.272224] LNet: Added LNI 192.168.206.121@tcp36 [8/256/0/180] [13057.715216] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp37 --if ens2 [13057.749851] LNet: Added LNI 192.168.206.121@tcp37 [8/256/0/180] [13064.677396] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp38 --if ens2 [13064.715479] LNet: Added LNI 192.168.206.121@tcp38 [8/256/0/180] [13072.256331] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp39 --if ens2 [13072.285678] LNet: Added LNI 192.168.206.121@tcp39 [8/256/0/180] [13079.166387] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp40 --if ens2 [13079.190765] LNet: Added LNI 192.168.206.121@tcp40 [8/256/0/180] [13086.156921] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp41 --if ens2 [13086.182218] LNet: Added LNI 192.168.206.121@tcp41 [8/256/0/180] [13093.813990] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp42 --if ens2 [13093.850404] LNet: Added LNI 192.168.206.121@tcp42 [8/256/0/180] [13101.175719] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp43 --if ens2 [13101.248969] LNet: Added LNI 192.168.206.121@tcp43 [8/256/0/180] [13109.625605] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp44 --if ens2 [13109.655744] LNet: Added LNI 192.168.206.121@tcp44 [8/256/0/180] [13116.985204] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp45 --if ens2 [13117.036682] LNet: Added LNI 192.168.206.121@tcp45 [8/256/0/180] [13123.971521] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp46 --if ens2 [13123.997722] LNet: Added LNI 192.168.206.121@tcp46 [8/256/0/180] [13132.027575] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp47 --if ens2 [13132.075581] LNet: Added LNI 192.168.206.121@tcp47 [8/256/0/180] [13140.292629] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp48 --if ens2 [13140.324353] LNet: Added LNI 192.168.206.121@tcp48 [8/256/0/180] [13148.960621] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp49 --if ens2 [13148.991706] LNet: Added LNI 192.168.206.121@tcp49 [8/256/0/180] [13157.956439] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing /home/green/git/lustre-release/lustre/../lnet/utils/lnetctl net add --net tcp50 --if ens2 [13158.027497] LNet: Added LNI 192.168.206.121@tcp50 [8/256/0/180] [13176.289904] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13176.300944] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13176.329176] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13180.385234] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13180.391318] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13180.397458] Lustre: Skipped 1 previous similar message [13180.405484] Lustre: Skipped 1 previous similar message [13182.770683] Lustre: server umount lustre-MDT0000 complete [13186.950408] LustreError: 473887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804038 with bad export cookie 13854143823917557813 [13186.951920] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13186.969637] LustreError: 473887:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 4 previous similar messages [13187.461553] Lustre: server umount lustre-MDT0001 complete [13205.983277] Lustre: 472367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788804041/real 1788804041] req@00000000bb8b88a4 x1875696586757888/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788804057 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13206.037783] Lustre: lustre-MDT0001-lwp-OST0000: Connection to lustre-MDT0001 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13207.009350] Lustre: 472366:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788804042/real 1788804042] req@0000000061bcd2c1 x1875696586758016/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788804058 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13207.209399] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [13211.103094] Lustre: 472365:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788804046/real 1788804046] req@000000007d55e467 x1875696586758144/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788804062 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13213.222194] Lustre: 472365:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788804048/real 1788804048] req@00000000e8a7d728 x1875696586758400/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788804064 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13216.729970] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13218.271348] Lustre: 472367:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1788804053/real 1788804053] req@00000000682bb9e4 x1875696586758528/t0(0) o400->lustre-MDT0001-lwp-OST0000@0@lo:12/10 lens 224/224 e 0 to 1 dl 1788804069 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13232.671862] LustreError: 472363:0:(client.c:1404:ptlrpc_import_delay_req()) @@@ invalidate in flight req@00000000e8a7d728 x1875696586759296/t0(0) o250->MGC192.168.206.121@tcp@0@lo:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl Rpc:NQU/200/ffffffff rc 0/-1 job:'' uid:0 gid:0 projid:4294967295 [13232.984383] LustreError: 475052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13233.007274] LustreError: 475052:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 4 previous similar messages [13233.045645] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13235.111269] LustreError: 475053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13235.115758] Lustre: lustre-MDT0000-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13235.132285] LustreError: 475053:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13236.128670] LustreError: 475052:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13236.772969] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13241.326414] LustreError: 475053:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13245.173688] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13245.529955] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13250.349820] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13251.573403] Lustre: lustre-MDT0001-lwp-OST0000: Connection restored to 0@lo (at 0@lo) [13258.678509] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13265.429851] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13291.491104] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13291.508724] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13291.537377] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13291.540505] Lustre: Skipped 1 previous similar message [13295.839602] Lustre: server umount lustre-OST0000 complete [13301.729419] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13301.740940] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13301.757476] Lustre: Skipped 1 previous similar message [13301.765111] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13301.768961] Lustre: Skipped 2 previous similar messages [13306.764613] Lustre: server umount lustre-MDT0000 complete [13307.874672] LustreError: 499318:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13307.897290] LustreError: 499318:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13311.261086] LustreError: 473085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804162 with bad export cookie 13854143823917560403 [13311.269178] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13311.290995] LustreError: 473085:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 3 previous similar messages [13311.810619] Lustre: server umount lustre-MDT0001 complete [13325.893983] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [13329.072920] Key type lgssc unregistered [13329.547833] LNet: 502118:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13329.554522] LNetError: 502118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13330.591304] LNet: Removed LNI 192.168.206.121@tcp [13330.593421] LNetError: 502118:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13330.599868] LNet: Removed LNI 192.168.206.121@tcp10 [13330.608633] LNet: Removed LNI 192.168.206.121@tcp11 [13330.613468] LNet: Removed LNI 192.168.206.121@tcp12 [13330.618845] LNet: Removed LNI 192.168.206.121@tcp13 [13330.626416] LNet: Removed LNI 192.168.206.121@tcp14 [13330.632900] LNet: Removed LNI 192.168.206.121@tcp15 [13330.638065] LNet: Removed LNI 192.168.206.121@tcp16 [13330.641304] LNet: Removed LNI 192.168.206.121@tcp17 [13330.643175] LNet: Removed LNI 192.168.206.121@tcp18 [13330.651878] LNet: Removed LNI 192.168.206.121@tcp19 [13330.659380] LNet: Removed LNI 192.168.206.121@tcp20 [13330.669196] LNet: Removed LNI 192.168.206.121@tcp21 [13330.672825] LNet: Removed LNI 192.168.206.121@tcp22 [13330.676301] LNet: Removed LNI 192.168.206.121@tcp23 [13330.678707] LNet: Removed LNI 192.168.206.121@tcp24 [13330.681443] LNet: Removed LNI 192.168.206.121@tcp25 [13330.686710] LNet: Removed LNI 192.168.206.121@tcp26 [13330.689887] LNet: Removed LNI 192.168.206.121@tcp27 [13330.696250] LNet: Removed LNI 192.168.206.121@tcp28 [13330.699399] LNet: Removed LNI 192.168.206.121@tcp29 [13330.703622] LNet: Removed LNI 192.168.206.121@tcp30 [13330.709364] LNet: Removed LNI 192.168.206.121@tcp31 [13330.712634] LNet: Removed LNI 192.168.206.121@tcp32 [13330.715312] LNet: Removed LNI 192.168.206.121@tcp33 [13330.718270] LNet: Removed LNI 192.168.206.121@tcp34 [13330.721567] LNet: Removed LNI 192.168.206.121@tcp35 [13330.730469] LNet: Removed LNI 192.168.206.121@tcp36 [13330.732742] LNet: Removed LNI 192.168.206.121@tcp37 [13330.735839] LNet: Removed LNI 192.168.206.121@tcp38 [13330.738429] LNet: Removed LNI 192.168.206.121@tcp39 [13330.741141] LNet: Removed LNI 192.168.206.121@tcp40 [13330.743457] LNet: Removed LNI 192.168.206.121@tcp41 [13330.746163] LNet: Removed LNI 192.168.206.121@tcp42 [13330.748178] LNet: Removed LNI 192.168.206.121@tcp43 [13330.751072] LNet: Removed LNI 192.168.206.121@tcp44 [13330.753649] LNet: Removed LNI 192.168.206.121@tcp45 [13330.755748] LNet: Removed LNI 192.168.206.121@tcp46 [13330.765255] LNet: Removed LNI 192.168.206.121@tcp47 [13330.773597] LNet: Removed LNI 192.168.206.121@tcp48 [13330.777597] LNet: Removed LNI 192.168.206.121@tcp49 [13330.785968] LNet: Removed LNI 192.168.206.121@tcp50 [13331.867682] Key type .llcrypt unregistered [13331.880545] Key type ._llcrypt unregistered [13355.298077] Key type ._llcrypt registered [13355.302555] Key type .llcrypt registered [13355.461161] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [13371.221962] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [13372.223152] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13372.238177] alg: No test for adler32 (adler32-zlib) [13373.358457] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [13373.689429] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [13375.407264] Key type lgssc registered [13376.592936] Lustre: Echo OBD driver; http://www.lustre.org/ [13412.708426] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13412.730411] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13413.960109] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13413.990869] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13414.099619] Lustre: lustre-MDT0000: new disk, initializing [13414.248378] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13414.287326] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13419.326152] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13430.116691] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13430.260337] Lustre: 505326:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13430.353852] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13430.357981] Lustre: Skipped 1 previous similar message [13430.440583] Lustre: lustre-MDT0001: new disk, initializing [13430.546056] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13430.595851] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13430.615848] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13435.459623] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13445.795058] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13452.477217] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13461.136655] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13461.411673] Lustre: *** cfs_fail_loc=90f, val=0*** [13461.559659] Lustre: lustre-OST0000: new disk, initializing [13461.569898] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13461.589408] Lustre: 507344:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13461.739384] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13468.234802] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13468.243954] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13468.388757] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13468.420384] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13478.248690] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13489.124200] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13489.125385] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13489.139790] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13489.148928] Lustre: Skipped 1 previous similar message [13489.164447] Lustre: Skipped 1 previous similar message [13492.758973] Lustre: server umount lustre-OST0000 complete [13499.361602] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13499.365304] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13499.374914] Lustre: Skipped 1 previous similar message [13499.404741] Lustre: Skipped 1 previous similar message [13504.382343] Lustre: server umount lustre-MDT0000 complete [13508.170870] LustreError: 505320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804359 with bad export cookie 2685988779988470821 [13508.172981] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13508.180330] LustreError: 505320:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 2 previous similar messages [13508.643909] Lustre: server umount lustre-MDT0001 complete [13520.981883] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [13523.792199] Key type lgssc unregistered [13524.168602] LNet: 509252:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13524.181644] LNetError: 509252:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13524.208740] LNet: Removed LNI 192.168.206.121@tcp [13525.099687] Key type .llcrypt unregistered [13525.101380] Key type ._llcrypt unregistered [13545.199649] Key type ._llcrypt registered [13545.202468] Key type .llcrypt registered [13545.327876] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [13546.345694] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13546.488430] alg: No test for adler32 (adler32-zlib) [13547.623296] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [13547.951604] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [13549.647181] Key type lgssc registered [13550.871597] Lustre: Echo OBD driver; http://www.lustre.org/ [13559.350223] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13559.375264] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13560.877958] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13565.520477] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13570.021086] LustreError: 510523:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13570.033898] LustreError: 510523:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 2 previous similar messages [13574.905860] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13575.136541] LustreError: 510524:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0001: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13575.345457] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13580.089768] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13588.708508] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13595.018191] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13601.525250] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13601.864845] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13607.678695] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13614.658755] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13629.482746] loop0: detected capacity change from 0 to 204800000 [13629.616804] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13629.632361] blk_update_request: operation not supported error, dev loop0, sector 208 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13629.644850] blk_update_request: operation not supported error, dev loop0, sector 160232 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13635.546074] loop0: detected capacity change from 0 to 204800000 [13635.764115] blk_update_request: operation not supported error, dev loop0, sector 399872 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13635.781958] blk_update_request: operation not supported error, dev loop0, sector 8224 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13635.797169] blk_update_request: operation not supported error, dev loop0, sector 58264 op 0x9:(WRITE_ZEROES) flags 0x400800 phys_seg 0 prio class 0 [13640.676537] loop0: detected capacity change from 0 to 204800000 [13646.361335] LDISKFS-fs (dm-4): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13646.524619] Lustre: fs15246-OST0000: new disk, initializing [13646.529743] Lustre: srv-fs15246-OST0000: No data found on store. Initialize space. [13646.535950] Lustre: 514119:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13646.588822] Lustre: fs15246-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13652.202579] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13661.578290] loop1: detected capacity change from 0 to 204800000 [13667.057940] LDISKFS-fs (dm-5): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13667.134291] Lustre: Setting parameter fs15246-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log fs15246-MDT0000 [13667.171295] Lustre: ctl-fs15246-MDT0000: No data found on store. Initialize space. [13667.230874] Lustre: fs15246-MDT0000: new disk, initializing [13667.378899] Lustre: fs15246-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13667.407552] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13667.509127] Lustre: srv-fs15246-OST0000: Waiting to contact MDT0000 to allocate super-sequence: rc = -115 [13668.580113] Lustre: 515240:0:(client.c:1652:after_reply()) @@@ resending request on EINPROGRESS req@000000000302145d x1875697360483328/t0(0) o700->fs15246-OST0000-osc-MDT0000@0@lo:31/4 lens 264/248 e 0 to 0 dl 1788804535 ref 2 fl Rpc:RQU/202/0 rc 0/-115 job:'' uid:0 gid:0 projid:4294967295 [13672.264601] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13677.665197] Lustre: ctl-fs15246-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:0:ost [13677.676308] Lustre: cli-fs15246-OST0000-super: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:0:ost] [13677.725410] Lustre: fs15246-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x240000400 [13694.952782] Lustre: fs15246-MDT0000-lwp-OST0000: Connection to fs15246-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13694.977682] Lustre: fs15246-MDT0000: Not available for connect from 0@lo (stopping) [13699.829904] Lustre: server umount fs15246-MDT0000 complete [13700.076063] LustreError: 510519:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13705.210934] LustreError: 510520:0:(ldlm_lib.c:1190:target_handle_connect()) fs15246-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13709.718225] Lustre: server umount fs15246-OST0000 complete [13723.107345] LustreError: lustre-OST0000-osc-MDT0001: operation ost_statfs to node 0@lo failed: rc = -107 [13723.108656] Lustre: lustre-OST0000-osc-MDT0000: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13723.112661] LustreError: Skipped 1 previous similar message [13723.117041] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13723.131483] Lustre: Skipped 1 previous similar message [13723.149300] Lustre: Skipped 1 previous similar message [13725.676834] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13728.175813] Lustre: server umount lustre-OST0000 complete [13734.368238] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13734.375186] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13734.394446] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13734.399134] Lustre: Skipped 1 previous similar message [13738.580461] Lustre: server umount lustre-MDT0000 complete [13741.029245] LustreError: 511263:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13741.043435] LustreError: 511263:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 1 previous similar message [13742.195444] LustreError: 511257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804593 with bad export cookie 2306260485324685730 [13742.197414] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13742.206317] LustreError: 511257:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13742.706248] Lustre: server umount lustre-MDT0001 complete [13754.094516] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [13756.654351] Key type lgssc unregistered [13756.964758] LNet: 518344:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [13756.980796] LNetError: 518344:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [13757.003378] LNet: Removed LNI 192.168.206.121@tcp [13757.829692] Key type .llcrypt unregistered [13757.836119] Key type ._llcrypt unregistered [13776.370353] Key type ._llcrypt registered [13776.375091] Key type .llcrypt registered [13776.509995] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [13777.565440] libcfs: HW NUMA nodes: 1, HW CPU cores: 4, npartitions: 1 [13777.587274] alg: No test for adler32 (adler32-zlib) [13778.663318] Lustre: Lustre: Build Version: 2.17.57_82_gfdb2c0c [13778.893759] LNet: Added LNI 192.168.206.121@tcp [8/256/0/180] [13780.567176] Key type lgssc registered [13781.794665] Lustre: Echo OBD driver; http://www.lustre.org/ [13819.602736] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [13831.046593] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [13868.998579] Lustre: lustre-MDT0000: mounting server target with '-t lustre' deprecated, use '-t lustre_tgt' [13869.018415] LDISKFS-fs (dm-0): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13870.202923] Lustre: Setting parameter lustre-MDT0000.mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity in log lustre-MDT0000 [13870.235665] Lustre: ctl-lustre-MDT0000: No data found on store. Initialize space. [13870.293503] Lustre: lustre-MDT0000: new disk, initializing [13870.351852] Lustre: lustre-MDT0000: Imperative Recovery not enabled, recovery window 60-180 [13870.369101] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000200000400-0x0000000240000400]:0:mdt [13874.514864] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13884.419288] LDISKFS-fs (dm-1): mounted filesystem with ordered data mode. Opts: user_xattr,errors=remount-ro,no_mbcache,nodelalloc [13884.496368] Lustre: 522813:0:(mgs_llog.c:1450:mgs_modify_param()) MGS: modify lustre-MDT0001/mdt.identity_upcall=/home/green/git/lustre-release/lustre/utils/l_getidentity (mode = 0) failed: rc = -17 [13884.580883] Lustre: srv-lustre-MDT0001: No data found on store. Initialize space. [13884.585519] Lustre: Skipped 1 previous similar message [13884.731903] Lustre: lustre-MDT0001: new disk, initializing [13884.825366] Lustre: lustre-MDT0001: Imperative Recovery not enabled, recovery window 60-180 [13884.860981] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000240000400-0x0000000280000400]:1:mdt [13884.876517] Lustre: cli-ctl-lustre-MDT0001: Allocated super-sequence [0x0000000240000400-0x0000000280000400]:1:mdt] [13889.009176] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13899.529256] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0000-mdc-*.mds_server_uuid 1475 0 [13905.450380] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount FULL mdc.lustre-MDT0001-mdc-*.mds_server_uuid 1475 0 [13912.718547] LDISKFS-fs (dm-2): mounted filesystem with ordered data mode. Opts: errors=remount-ro,no_mbcache,nodelalloc [13913.034691] Lustre: lustre-OST0000: new disk, initializing [13913.037971] Lustre: srv-lustre-OST0000: No data found on store. Initialize space. [13913.044503] Lustre: 524781:0:(osd_compat.c:1353:osd_object_spec_find()) UNKNOWN COMPAT FID [0x200000001:0x101e:0x0] [13913.147329] Lustre: lustre-OST0000: Imperative Recovery not enabled, recovery window 60-180 [13918.484689] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_default_debug -1 all [13921.855987] Lustre: ctl-lustre-MDT0000: super-sequence allocation rc = 0 [0x0000000280000400-0x00000002c0000400]:0:ost [13921.866358] Lustre: cli-lustre-OST0000-super: Allocated super-sequence [0x0000000280000400-0x00000002c0000400]:0:ost] [13921.952090] Lustre: lustre-OST0000-osc-MDT0000: update sequence from 0x100000000 to 0x280000401 [13927.089367] Lustre: DEBUG MARKER: oleg621-client.virtnet: executing wait_import_state_mount (FULL|IDLE) osc.lustre-OST0000-osc-[-0-9a-f]*.ost_server_uuid 1475 0 [13931.784048] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid 50 [13932.071475] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0000.ost_server_uuid in FULL state after 0 sec [13937.571687] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing wait_import_state FULL os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid 50 [13937.822843] Lustre: DEBUG MARKER: os[cp].lustre-OST0000-osc-MDT0001.ost_server_uuid in FULL state after 0 sec [13942.758302] LustreError: lustre-OST0000-osc-MDT0000: operation ost_statfs to node 0@lo failed: rc = -107 [13942.759599] Lustre: lustre-OST0000-osc-MDT0001: Connection to lustre-OST0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13942.763404] LustreError: Skipped 1 previous similar message [13942.766384] Lustre: lustre-OST0000: Not available for connect from 0@lo (stopping) [13942.770932] Lustre: Skipped 1 previous similar message [13942.788909] Lustre: Skipped 1 previous similar message [13946.219426] Lustre: server umount lustre-OST0000 complete [13951.456639] LustreError: lustre-MDT0000-osp-MDT0001: operation mds_statfs to node 0@lo failed: rc = -107 [13951.468134] Lustre: lustre-MDT0000-osp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13951.486774] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13952.994751] Lustre: lustre-MDT0000: Not available for connect from 0@lo (stopping) [13952.994933] Lustre: lustre-MDT0000-lwp-MDT0001: Connection to lustre-MDT0000 (at 0@lo) was lost; in progress operations using this service will wait for recovery to complete [13956.825933] Lustre: server umount lustre-MDT0000 complete [13958.114528] LustreError: 522820:0:(ldlm_lib.c:1190:target_handle_connect()) lustre-MDT0000: not available for connect from 0@lo (no target). If you are running an HA pair check that the target is mounted on the other server. [13958.137281] LustreError: 522820:0:(ldlm_lib.c:1190:target_handle_connect()) Skipped 3 previous similar messages [13960.932291] LustreError: 522806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) ldlm_cancel from 0@lo arrived at 1788804811 with bad export cookie 12661605814152341728 [13960.935482] LustreError: MGC192.168.206.121@tcp: Connection to MGS (at 0@lo) was lost; in progress operations using this service will fail [13960.953363] LustreError: 522806:0:(ldlm_lockd.c:2564:ldlm_cancel_handler()) Skipped 1 previous similar message [13961.483804] Lustre: server umount lustre-MDT0001 complete [13986.722348] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing set_hostid [13996.317500] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing load_modules_local [14025.343723] Lustre: DEBUG MARKER: == conf-sanity test complete, duration 13880 sec ========= 14:14:35 (1788804875) [14026.931967] Lustre: DEBUG MARKER: === conf-sanity: start cleanup 14:14:37 (1788804877) === [14030.749616] Lustre: DEBUG MARKER: === conf-sanity: finish cleanup 14:14:41 (1788804881) === [14069.533855] Lustre: DEBUG MARKER: oleg621-server.virtnet: executing unload_modules_local [14072.510381] Key type lgssc unregistered [14072.772384] LNet: 531428:0:(lib-ptl.c:964:lnet_clear_lazy_portal()) Active lazy portal 0 on exit [14072.787456] LNetError: 531428:0:(acceptor.c:252:lnet_acceptor_remove_socket()) Interface ens2 not found [14072.813109] LNet: Removed LNI 192.168.206.121@tcp [14073.659445] Key type .llcrypt unregistered [14073.664369] Key type ._llcrypt unregistered